Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260810 08:00:07.036031 14567 test_util.cc:268] Using random seed: -1105814531
2026-08-10T08:00:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T08:00:07Z Disabled control of system clock
I20260810 08:00:07.096151 14567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzwAODw/build/tsan/bin/kudu
/tmp/dist-test-taskzwAODw/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.57.254:43757
--webserver_interface=127.14.57.254
--webserver_port=0
--builtin_ntp_servers=127.14.57.212:34009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.57.254:43757,127.14.57.253:40439,127.14.57.252:33819
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260810 08:00:07.442356 14578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 08:00:07.443054 14578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 08:00:07.443545 14578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 08:00:07.475046 14578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 08:00:07.475401 14578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 08:00:07.475706 14578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 08:00:07.475958 14578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 08:00:07.512035 14578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.57.212:34009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.14.57.254:43757,127.14.57.253:40439,127.14.57.252:33819
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.57.254:43757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.57.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 07:02:49 UTC on ccycloud.quasar-ojudvd.root.comops.site
build id 4577116
TSAN enabled
I20260810 08:00:07.513576 14578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 08:00:07.515543 14578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 08:00:07.544286 14587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 08:00:07.544307 14584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 08:00:07.544306 14585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 08:00:07.544866 14578 server_base.cc:1034] running on GCE node
I20260810 08:00:08.873962 14578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 08:00:08.877215 14578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 08:00:08.878701 14578 hybrid_clock.cc:648] HybridClock initialized: now 1786348808878558 us; error 143 us; skew 500 ppm
I20260810 08:00:08.880113 14578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 08:00:08.892910 14578 webserver.cc:460] Webserver started at http://127.14.57.254:32825/ using document root <none> and password file <none>
I20260810 08:00:08.894382 14578 fs_manager.cc:359] Metadata directory not provided
I20260810 08:00:08.894622 14578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 08:00:08.895154 14578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 08:00:08.901561 14578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-0/data/instance:
uuid: "99f2d868d76c440f8443b33e1cd19f58"
format_stamp: "Formatted at 2026-08-10 08:00:08 on dist-test-slave-p3t5"
I20260810 08:00:08.903748 14578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-0/wal/instance:
uuid: "99f2d868d76c440f8443b33e1cd19f58"
format_stamp: "Formatted at 2026-08-10 08:00:08 on dist-test-slave-p3t5"
I20260810 08:00:08.913952 14578 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.007s sys 0.004s
I20260810 08:00:08.921869 14594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 08:00:08.923508 14578 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260810 08:00:08.923920 14578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-0/data,/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-0/wal
uuid: "99f2d868d76c440f8443b33e1cd19f58"
format_stamp: "Formatted at 2026-08-10 08:00:08 on dist-test-slave-p3t5"
I20260810 08:00:08.924312 14578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 08:00:08.976548 14578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 08:00:08.978349 14578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 08:00:08.978922 14578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 08:00:09.059808 14578 rpc_server.cc:307] RPC server started. Bound to: 127.14.57.254:43757
I20260810 08:00:09.059935 14649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.57.254:43757 every 8 connection(s)
I20260810 08:00:09.062971 14578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-0/data/info.pb
I20260810 08:00:09.064633 14567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzwAODw/build/tsan/bin/kudu as pid 14578
I20260810 08:00:09.065124 14567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-0/wal/instance
I20260810 08:00:09.069861 14567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzwAODw/build/tsan/bin/kudu
/tmp/dist-test-taskzwAODw/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.57.253:40439
--webserver_interface=127.14.57.253
--webserver_port=0
--builtin_ntp_servers=127.14.57.212:34009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.57.254:43757,127.14.57.253:40439,127.14.57.252:33819
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260810 08:00:09.072101 14650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:09.085695 14650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } has no permanent_uuid. Determining permanent_uuid...
I20260810 08:00:09.126837 14650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } has no permanent_uuid. Determining permanent_uuid...
W20260810 08:00:09.129021 14597 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.57.253:40439: connect: Connection refused (error 111)
W20260810 08:00:09.133484 14650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.57.253:40439: Network error: Client connection negotiation failed: client connection to 127.14.57.253:40439: connect: Connection refused (error 111)
I20260810 08:00:09.183143 14650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } attempt: 1
W20260810 08:00:09.187465 14650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.57.253:40439: Network error: Client connection negotiation failed: client connection to 127.14.57.253:40439: connect: Connection refused (error 111)
I20260810 08:00:09.256000 14650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } attempt: 2
W20260810 08:00:09.259964 14650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.57.253:40439: Network error: Client connection negotiation failed: client connection to 127.14.57.253:40439: connect: Connection refused (error 111)
I20260810 08:00:09.351475 14650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } attempt: 3
W20260810 08:00:09.355610 14650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.57.253:40439: Network error: Client connection negotiation failed: client connection to 127.14.57.253:40439: connect: Connection refused (error 111)
W20260810 08:00:09.393600 14654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 08:00:09.394333 14654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 08:00:09.394840 14654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 08:00:09.426482 14654 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 08:00:09.426880 14654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 08:00:09.427151 14654 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 08:00:09.427404 14654 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 08:00:09.463004 14654 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.57.212:34009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.14.57.254:43757,127.14.57.253:40439,127.14.57.252:33819
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.57.253:40439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.57.253
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 07:02:49 UTC on ccycloud.quasar-ojudvd.root.comops.site
build id 4577116
TSAN enabled
I20260810 08:00:09.464377 14654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 08:00:09.466089 14654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 08:00:09.483448 14663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 08:00:09.499111 14650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } attempt: 4
W20260810 08:00:09.503044 14650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.57.253:40439: Network error: Client connection negotiation failed: client connection to 127.14.57.253:40439: connect: Connection refused (error 111)
I20260810 08:00:09.802798 14650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } attempt: 5
W20260810 08:00:09.808283 14650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.57.253:40439: Network error: Client connection negotiation failed: client connection to 127.14.57.253:40439: connect: Connection refused (error 111)
I20260810 08:00:10.355744 14650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } attempt: 6
W20260810 08:00:10.364183 14650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.57.253:40439: Network error: Client connection negotiation failed: client connection to 127.14.57.253:40439: connect: Connection refused (error 111)
W20260810 08:00:09.483451 14664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 08:00:09.484578 14666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 08:00:10.759522 14665 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260810 08:00:10.759675 14654 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260810 08:00:10.763586 14654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 08:00:10.766927 14654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 08:00:10.768412 14654 hybrid_clock.cc:648] HybridClock initialized: now 1786348810768371 us; error 64 us; skew 500 ppm
I20260810 08:00:10.769395 14654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 08:00:10.776605 14654 webserver.cc:460] Webserver started at http://127.14.57.253:43877/ using document root <none> and password file <none>
I20260810 08:00:10.777696 14654 fs_manager.cc:359] Metadata directory not provided
I20260810 08:00:10.777935 14654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 08:00:10.778434 14654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 08:00:10.783694 14654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-1/data/instance:
uuid: "38a7164890624423a7427871aa2b10e0"
format_stamp: "Formatted at 2026-08-10 08:00:10 on dist-test-slave-p3t5"
I20260810 08:00:10.784933 14654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-1/wal/instance:
uuid: "38a7164890624423a7427871aa2b10e0"
format_stamp: "Formatted at 2026-08-10 08:00:10 on dist-test-slave-p3t5"
I20260810 08:00:10.792814 14654 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.005s
I20260810 08:00:10.799706 14676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 08:00:10.801026 14654 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260810 08:00:10.801415 14654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-1/data,/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-1/wal
uuid: "38a7164890624423a7427871aa2b10e0"
format_stamp: "Formatted at 2026-08-10 08:00:10 on dist-test-slave-p3t5"
I20260810 08:00:10.801873 14654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 08:00:10.837702 14654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 08:00:10.839344 14654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 08:00:10.839874 14654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 08:00:10.918584 14654 rpc_server.cc:307] RPC server started. Bound to: 127.14.57.253:40439
I20260810 08:00:10.918680 14732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.57.253:40439 every 8 connection(s)
I20260810 08:00:10.921397 14654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-1/data/info.pb
I20260810 08:00:10.923388 14567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzwAODw/build/tsan/bin/kudu as pid 14654
I20260810 08:00:10.923945 14567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-1/wal/instance
I20260810 08:00:10.926978 14567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzwAODw/build/tsan/bin/kudu
/tmp/dist-test-taskzwAODw/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.57.252:33819
--webserver_interface=127.14.57.252
--webserver_port=0
--builtin_ntp_servers=127.14.57.212:34009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.57.254:43757,127.14.57.253:40439,127.14.57.252:33819
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260810 08:00:10.928181 14733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:10.941285 14733 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } has no permanent_uuid. Determining permanent_uuid...
I20260810 08:00:10.966427 14733 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } has no permanent_uuid. Determining permanent_uuid...
I20260810 08:00:10.984690 14733 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } has no permanent_uuid. Determining permanent_uuid...
W20260810 08:00:10.987403 14681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.57.252:33819: connect: Connection refused (error 111)
W20260810 08:00:10.992349 14733 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.57.252:33819: Network error: Client connection negotiation failed: client connection to 127.14.57.252:33819: connect: Connection refused (error 111)
I20260810 08:00:11.041981 14733 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } attempt: 1
W20260810 08:00:11.046512 14733 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.57.252:33819: Network error: Client connection negotiation failed: client connection to 127.14.57.252:33819: connect: Connection refused (error 111)
I20260810 08:00:11.115002 14733 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } attempt: 2
W20260810 08:00:11.119261 14733 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.57.252:33819: Network error: Client connection negotiation failed: client connection to 127.14.57.252:33819: connect: Connection refused (error 111)
I20260810 08:00:11.210775 14733 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } attempt: 3
W20260810 08:00:11.215050 14733 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.57.252:33819: Network error: Client connection negotiation failed: client connection to 127.14.57.252:33819: connect: Connection refused (error 111)
W20260810 08:00:11.255105 14735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 08:00:11.255795 14735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 08:00:11.256286 14735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 08:00:11.288465 14735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 08:00:11.288812 14735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 08:00:11.289125 14735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 08:00:11.289388 14735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 08:00:11.325661 14735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.57.212:34009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.14.57.254:43757,127.14.57.253:40439,127.14.57.252:33819
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.57.252:33819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.57.252
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 07:02:49 UTC on ccycloud.quasar-ojudvd.root.comops.site
build id 4577116
TSAN enabled
I20260810 08:00:11.327035 14735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 08:00:11.328696 14735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 08:00:11.346982 14746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 08:00:11.358507 14733 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } attempt: 4
W20260810 08:00:11.362926 14733 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.57.252:33819: Network error: Client connection negotiation failed: client connection to 127.14.57.252:33819: connect: Connection refused (error 111)
I20260810 08:00:11.424635 14650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } attempt: 7
I20260810 08:00:11.444955 14650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } has no permanent_uuid. Determining permanent_uuid...
W20260810 08:00:11.452021 14650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.57.252:33819: Network error: Client connection negotiation failed: client connection to 127.14.57.252:33819: connect: Connection refused (error 111)
I20260810 08:00:11.510957 14650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } attempt: 1
W20260810 08:00:11.515524 14650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.57.252:33819: Network error: Client connection negotiation failed: client connection to 127.14.57.252:33819: connect: Connection refused (error 111)
I20260810 08:00:11.597240 14650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } attempt: 2
W20260810 08:00:11.601493 14650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.57.252:33819: Network error: Client connection negotiation failed: client connection to 127.14.57.252:33819: connect: Connection refused (error 111)
I20260810 08:00:11.662405 14733 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } attempt: 5
W20260810 08:00:11.666826 14733 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.57.252:33819: Network error: Client connection negotiation failed: client connection to 127.14.57.252:33819: connect: Connection refused (error 111)
I20260810 08:00:11.687150 14650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } attempt: 3
W20260810 08:00:11.691402 14650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.57.252:33819: Network error: Client connection negotiation failed: client connection to 127.14.57.252:33819: connect: Connection refused (error 111)
I20260810 08:00:11.832082 14650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } attempt: 4
W20260810 08:00:11.836558 14650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.57.252:33819: Network error: Client connection negotiation failed: client connection to 127.14.57.252:33819: connect: Connection refused (error 111)
I20260810 08:00:12.120241 14650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } attempt: 5
W20260810 08:00:12.124881 14650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.57.252:33819: Network error: Client connection negotiation failed: client connection to 127.14.57.252:33819: connect: Connection refused (error 111)
I20260810 08:00:12.214493 14733 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } attempt: 6
W20260810 08:00:12.222289 14733 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.57.252:33819: Network error: Client connection negotiation failed: client connection to 127.14.57.252:33819: connect: Connection refused (error 111)
W20260810 08:00:11.348055 14745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 08:00:11.349747 14735 server_base.cc:1034] running on GCE node
W20260810 08:00:11.347301 14749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 08:00:12.584824 14735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 08:00:12.587613 14735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 08:00:12.589011 14735 hybrid_clock.cc:648] HybridClock initialized: now 1786348812588969 us; error 56 us; skew 500 ppm
I20260810 08:00:12.589911 14735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 08:00:12.596347 14735 webserver.cc:460] Webserver started at http://127.14.57.252:34789/ using document root <none> and password file <none>
I20260810 08:00:12.597409 14735 fs_manager.cc:359] Metadata directory not provided
I20260810 08:00:12.597715 14735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 08:00:12.598212 14735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 08:00:12.603197 14735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-2/data/instance:
uuid: "1e8cadb26fe0428a964f66ef8172f3be"
format_stamp: "Formatted at 2026-08-10 08:00:12 on dist-test-slave-p3t5"
I20260810 08:00:12.604444 14735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-2/wal/instance:
uuid: "1e8cadb26fe0428a964f66ef8172f3be"
format_stamp: "Formatted at 2026-08-10 08:00:12 on dist-test-slave-p3t5"
I20260810 08:00:12.612054 14735 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260810 08:00:12.617983 14760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 08:00:12.619174 14735 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260810 08:00:12.619540 14735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-2/data,/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-2/wal
uuid: "1e8cadb26fe0428a964f66ef8172f3be"
format_stamp: "Formatted at 2026-08-10 08:00:12 on dist-test-slave-p3t5"
I20260810 08:00:12.619916 14735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 08:00:12.666330 14735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 08:00:12.667920 14735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 08:00:12.668416 14735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 08:00:12.677567 14650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } attempt: 6
W20260810 08:00:12.685140 14650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.57.252:33819: Network error: Client connection negotiation failed: client connection to 127.14.57.252:33819: connect: Connection refused (error 111)
I20260810 08:00:12.745896 14735 rpc_server.cc:307] RPC server started. Bound to: 127.14.57.252:33819
I20260810 08:00:12.745994 14817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.57.252:33819 every 8 connection(s)
I20260810 08:00:12.748903 14735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-2/data/info.pb
I20260810 08:00:12.755564 14818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:12.758000 14567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzwAODw/build/tsan/bin/kudu as pid 14735
I20260810 08:00:12.758524 14567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-2/wal/instance
I20260810 08:00:12.773490 14818 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } has no permanent_uuid. Determining permanent_uuid...
I20260810 08:00:12.810526 14818 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } has no permanent_uuid. Determining permanent_uuid...
I20260810 08:00:12.828223 14818 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } has no permanent_uuid. Determining permanent_uuid...
I20260810 08:00:12.848173 14818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be: Bootstrap starting.
I20260810 08:00:12.855557 14818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:12.858292 14818 log.cc:824] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be: Log is configured to *not* fsync() on all Append() calls
I20260810 08:00:12.863433 14818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be: No bootstrap required, opened a new log
I20260810 08:00:12.887089 14818 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:12.887749 14818 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:12.888085 14818 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e8cadb26fe0428a964f66ef8172f3be, State: Initialized, Role: FOLLOWER
I20260810 08:00:12.889050 14818 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:12.891856 14831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } } }
I20260810 08:00:12.892611 14831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [sys.catalog]: This master's current role is: FOLLOWER
I20260810 08:00:12.893420 14818 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [sys.catalog]: configured and running, proceeding with master startup.
W20260810 08:00:12.925894 14843 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 08:00:12.926559 14843 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260810 08:00:12.927614 14767 tablet.cc:2369] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 08:00:12.919695 14735 master_runner.cc:186] Error getting master registration for 127.14.57.254:43757: OK, instance_id { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" instance_seqno: 1786348809038252 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 08:00:12.955112 14735 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 08:00:13.129071 14831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:13.129737 14831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:13.132174 14831 leader_election.cc:288] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99f2d868d76c440f8443b33e1cd19f58 (127.14.57.254:43757), 38a7164890624423a7427871aa2b10e0 (127.14.57.253:40439)
I20260810 08:00:13.282948 14733 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } attempt: 7
I20260810 08:00:13.308704 14733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Bootstrap starting.
I20260810 08:00:13.314603 14733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:13.316562 14733 log.cc:824] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Log is configured to *not* fsync() on all Append() calls
I20260810 08:00:13.321400 14733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: No bootstrap required, opened a new log
I20260810 08:00:13.348235 14733 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:13.349035 14733 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:13.349301 14733 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38a7164890624423a7427871aa2b10e0, State: Initialized, Role: FOLLOWER
I20260810 08:00:13.350214 14733 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:13.353159 14847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } } }
I20260810 08:00:13.354066 14847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 08:00:13.355258 14733 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 08:00:13.356737 14705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e8cadb26fe0428a964f66ef8172f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38a7164890624423a7427871aa2b10e0" is_pre_election: true
I20260810 08:00:13.359719 14705 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e8cadb26fe0428a964f66ef8172f3be in term 0.
I20260810 08:00:13.361768 14764 leader_election.cc:302] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e8cadb26fe0428a964f66ef8172f3be, 38a7164890624423a7427871aa2b10e0; no voters:
I20260810 08:00:13.363027 14831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:13.363413 14831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:13.363816 14831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:13.372035 14831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:13.378441 14705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e8cadb26fe0428a964f66ef8172f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38a7164890624423a7427871aa2b10e0"
I20260810 08:00:13.379279 14705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:13.388435 14831 leader_election.cc:288] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [CANDIDATE]: Term 1 election: Requested vote from peers 99f2d868d76c440f8443b33e1cd19f58 (127.14.57.254:43757), 38a7164890624423a7427871aa2b10e0 (127.14.57.253:40439)
I20260810 08:00:13.394599 14705 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e8cadb26fe0428a964f66ef8172f3be in term 1.
I20260810 08:00:13.397120 14764 leader_election.cc:302] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e8cadb26fe0428a964f66ef8172f3be, 38a7164890624423a7427871aa2b10e0; no voters:
I20260810 08:00:13.398628 14831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [term 1 FOLLOWER]: Leader election won for term 1
I20260810 08:00:13.402968 14831 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [term 1 LEADER]: Becoming Leader. State: Replica: 1e8cadb26fe0428a964f66ef8172f3be, State: Running, Role: LEADER
I20260810 08:00:13.406226 14831 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:13.438417 14858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e8cadb26fe0428a964f66ef8172f3be. Latest consensus state: current_term: 1 leader_uuid: "1e8cadb26fe0428a964f66ef8172f3be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } } }
I20260810 08:00:13.439296 14858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [sys.catalog]: This master's current role is: LEADER
I20260810 08:00:13.449575 14863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 08:00:13.457468 14863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260810 08:00:13.459398 14861 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 08:00:13.459851 14861 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 08:00:13.477281 14705 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 1 FOLLOWER]: Refusing update from remote peer 1e8cadb26fe0428a964f66ef8172f3be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 08:00:13.479444 14858 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 08:00:13.421447 14654 master_runner.cc:186] Error getting master registration for 127.14.57.254:43757: OK, instance_id { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" instance_seqno: 1786348809038252 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 08:00:13.495548 14654 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 08:00:13.509398 14847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e8cadb26fe0428a964f66ef8172f3be. Latest consensus state: current_term: 1 leader_uuid: "1e8cadb26fe0428a964f66ef8172f3be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } } }
I20260810 08:00:13.510136 14847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 08:00:13.530107 14858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1e8cadb26fe0428a964f66ef8172f3be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } } }
I20260810 08:00:13.531032 14858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [sys.catalog]: This master's current role is: LEADER
I20260810 08:00:13.536336 14863 catalog_manager.cc:1347] Generated new cluster ID: eddaf5b0e44343438f86652258a1a0e1
I20260810 08:00:13.536701 14863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 08:00:13.560585 14847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1e8cadb26fe0428a964f66ef8172f3be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } } }
I20260810 08:00:13.561335 14847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 08:00:13.601166 14863 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 08:00:13.613760 14863 catalog_manager.cc:1504] Loading token signing keys...
I20260810 08:00:13.636499 14863 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be: Generated new TSK 0
I20260810 08:00:13.641431 14863 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 08:00:13.694749 14567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzwAODw/build/tsan/bin/kudu
/tmp/dist-test-taskzwAODw/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.57.193:0
--local_ip_for_outbound_sockets=127.14.57.193
--webserver_interface=127.14.57.193
--webserver_port=0
--tserver_master_addrs=127.14.57.254:43757,127.14.57.253:40439,127.14.57.252:33819
--builtin_ntp_servers=127.14.57.212:34009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260810 08:00:13.718565 14650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } attempt: 7
I20260810 08:00:13.759150 14650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58: Bootstrap starting.
I20260810 08:00:13.766613 14650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:13.769253 14650 log.cc:824] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58: Log is configured to *not* fsync() on all Append() calls
I20260810 08:00:13.780593 14650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58: No bootstrap required, opened a new log
I20260810 08:00:13.813177 14650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:13.814237 14650 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:13.814613 14650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f2d868d76c440f8443b33e1cd19f58, State: Initialized, Role: FOLLOWER
I20260810 08:00:13.815724 14650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:13.824088 14874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } } }
I20260810 08:00:13.827878 14874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 08:00:13.827718 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e8cadb26fe0428a964f66ef8172f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f2d868d76c440f8443b33e1cd19f58" is_pre_election: true
I20260810 08:00:13.827522 14650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 08:00:13.828681 14623 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e8cadb26fe0428a964f66ef8172f3be in term 0.
W20260810 08:00:13.829267 14623 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54362 (request call id 2) took 696 ms (client timeout 729 ms). Trace:
W20260810 08:00:13.829607 14623 rpcz_store.cc:269] 0810 08:00:13.132655 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:13.132840 (+ 185us) service_pool.cc:227] Handling call
0810 08:00:13.829198 (+696358us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 08:00:13.832345 14621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:13.837901 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e8cadb26fe0428a964f66ef8172f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f2d868d76c440f8443b33e1cd19f58"
I20260810 08:00:13.842363 14621 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 1 FOLLOWER]: Refusing update from remote peer 1e8cadb26fe0428a964f66ef8172f3be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 08:00:13.844079 14858 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 08:00:13.929010 14874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e8cadb26fe0428a964f66ef8172f3be. Latest consensus state: current_term: 1 leader_uuid: "1e8cadb26fe0428a964f66ef8172f3be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } } }
I20260810 08:00:13.929929 14874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 08:00:13.940979 14885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1e8cadb26fe0428a964f66ef8172f3be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } } }
I20260810 08:00:13.942107 14885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 08:00:13.954420 14890 catalog_manager.cc:1259] Loaded cluster ID: eddaf5b0e44343438f86652258a1a0e1
I20260810 08:00:13.965757 14890 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58: loading cluster ID for follower catalog manager: success
I20260810 08:00:13.990593 14890 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58: acquiring CA information for follower catalog manager: success
I20260810 08:00:13.996968 14890 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 08:00:14.299715 14870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 08:00:14.300436 14870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 08:00:14.301394 14870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 08:00:14.349102 14885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e8cadb26fe0428a964f66ef8172f3be)
I20260810 08:00:14.349655 14885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:14.351989 14885 leader_election.cc:288] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 38a7164890624423a7427871aa2b10e0 (127.14.57.253:40439), 1e8cadb26fe0428a964f66ef8172f3be (127.14.57.252:33819)
I20260810 08:00:14.354413 14705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99f2d868d76c440f8443b33e1cd19f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38a7164890624423a7427871aa2b10e0" is_pre_election: true
I20260810 08:00:14.355422 14791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99f2d868d76c440f8443b33e1cd19f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e8cadb26fe0428a964f66ef8172f3be" is_pre_election: true
I20260810 08:00:14.358074 14598 leader_election.cc:302] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99f2d868d76c440f8443b33e1cd19f58; no voters: 1e8cadb26fe0428a964f66ef8172f3be, 38a7164890624423a7427871aa2b10e0
I20260810 08:00:14.359454 14885 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 08:00:14.359566 14870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 08:00:14.361112 14870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.57.193
I20260810 08:00:14.425032 14870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.57.212:34009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.57.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.57.193
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.14.57.254:43757,127.14.57.253:40439,127.14.57.252:33819
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.57.193
--log_dir=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 07:02:49 UTC on ccycloud.quasar-ojudvd.root.comops.site
build id 4577116
TSAN enabled
I20260810 08:00:14.426891 14870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 08:00:14.429281 14870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 08:00:14.465595 14861 catalog_manager.cc:1259] Loaded cluster ID: eddaf5b0e44343438f86652258a1a0e1
I20260810 08:00:14.466040 14861 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: loading cluster ID for follower catalog manager: success
I20260810 08:00:14.474331 14861 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: acquiring CA information for follower catalog manager: success
I20260810 08:00:14.487133 14898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1e8cadb26fe0428a964f66ef8172f3be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } } }
I20260810 08:00:14.494345 14898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [sys.catalog]: This master's current role is: LEADER
I20260810 08:00:14.494587 14861 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 08:00:14.502919 14900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 08:00:14.526556 14906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e8cadb26fe0428a964f66ef8172f3be)
I20260810 08:00:14.527420 14906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:14.541101 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "38a7164890624423a7427871aa2b10e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99f2d868d76c440f8443b33e1cd19f58" is_pre_election: true
I20260810 08:00:14.545882 14906 leader_election.cc:288] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99f2d868d76c440f8443b33e1cd19f58 (127.14.57.254:43757), 1e8cadb26fe0428a964f66ef8172f3be (127.14.57.252:33819)
I20260810 08:00:14.546036 14791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "38a7164890624423a7427871aa2b10e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e8cadb26fe0428a964f66ef8172f3be" is_pre_election: true
I20260810 08:00:14.547644 14681 leader_election.cc:302] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38a7164890624423a7427871aa2b10e0; no voters: 1e8cadb26fe0428a964f66ef8172f3be, 99f2d868d76c440f8443b33e1cd19f58
I20260810 08:00:14.548625 14906 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 08:00:14.982381 14908 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e8cadb26fe0428a964f66ef8172f3be)
I20260810 08:00:14.983093 14908 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:14.986586 14791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99f2d868d76c440f8443b33e1cd19f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e8cadb26fe0428a964f66ef8172f3be" is_pre_election: true
I20260810 08:00:14.986860 14705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99f2d868d76c440f8443b33e1cd19f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38a7164890624423a7427871aa2b10e0" is_pre_election: true
I20260810 08:00:14.988373 14908 leader_election.cc:288] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 38a7164890624423a7427871aa2b10e0 (127.14.57.253:40439), 1e8cadb26fe0428a964f66ef8172f3be (127.14.57.252:33819)
I20260810 08:00:14.988705 14597 leader_election.cc:302] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99f2d868d76c440f8443b33e1cd19f58; no voters: 1e8cadb26fe0428a964f66ef8172f3be, 38a7164890624423a7427871aa2b10e0
I20260810 08:00:14.998670 14908 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 08:00:15.586140 14909 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e8cadb26fe0428a964f66ef8172f3be)
I20260810 08:00:15.594131 14909 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:15.599010 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "38a7164890624423a7427871aa2b10e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99f2d868d76c440f8443b33e1cd19f58" is_pre_election: true
I20260810 08:00:15.603253 14909 leader_election.cc:288] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99f2d868d76c440f8443b33e1cd19f58 (127.14.57.254:43757), 1e8cadb26fe0428a964f66ef8172f3be (127.14.57.252:33819)
I20260810 08:00:15.603828 14791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "38a7164890624423a7427871aa2b10e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e8cadb26fe0428a964f66ef8172f3be" is_pre_election: true
I20260810 08:00:15.605512 14681 leader_election.cc:302] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38a7164890624423a7427871aa2b10e0; no voters: 1e8cadb26fe0428a964f66ef8172f3be, 99f2d868d76c440f8443b33e1cd19f58
I20260810 08:00:15.606398 14909 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 08:00:16.168496 14910 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e8cadb26fe0428a964f66ef8172f3be)
I20260810 08:00:16.169189 14910 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:16.172632 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "38a7164890624423a7427871aa2b10e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99f2d868d76c440f8443b33e1cd19f58" is_pre_election: true
I20260810 08:00:16.175789 14791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "38a7164890624423a7427871aa2b10e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e8cadb26fe0428a964f66ef8172f3be" is_pre_election: true
I20260810 08:00:16.177349 14681 leader_election.cc:302] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38a7164890624423a7427871aa2b10e0; no voters: 1e8cadb26fe0428a964f66ef8172f3be, 99f2d868d76c440f8443b33e1cd19f58
I20260810 08:00:16.182988 14910 leader_election.cc:288] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99f2d868d76c440f8443b33e1cd19f58 (127.14.57.254:43757), 1e8cadb26fe0428a964f66ef8172f3be (127.14.57.252:33819)
I20260810 08:00:16.184334 14910 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 08:00:14.504233 14902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 08:00:14.513729 14897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 08:00:16.300355 14901 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1795 milliseconds
I20260810 08:00:16.300493 14870 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260810 08:00:16.302156 14870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 08:00:16.310782 14870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 08:00:16.312280 14870 hybrid_clock.cc:648] HybridClock initialized: now 1786348816312242 us; error 59 us; skew 500 ppm
I20260810 08:00:16.313441 14870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 08:00:16.341643 14870 webserver.cc:460] Webserver started at http://127.14.57.193:34441/ using document root <none> and password file <none>
I20260810 08:00:16.343169 14870 fs_manager.cc:359] Metadata directory not provided
I20260810 08:00:16.343494 14870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 08:00:16.344135 14870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 08:00:16.351919 14870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/data/instance:
uuid: "a91b27762182454baada71e91ceab224"
format_stamp: "Formatted at 2026-08-10 08:00:16 on dist-test-slave-p3t5"
I20260810 08:00:16.353647 14870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal/instance:
uuid: "a91b27762182454baada71e91ceab224"
format_stamp: "Formatted at 2026-08-10 08:00:16 on dist-test-slave-p3t5"
I20260810 08:00:16.371531 14870 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.009s sys 0.006s
I20260810 08:00:16.386677 14917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 08:00:16.388109 14870 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.008s sys 0.000s
I20260810 08:00:16.388520 14870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal
uuid: "a91b27762182454baada71e91ceab224"
format_stamp: "Formatted at 2026-08-10 08:00:16 on dist-test-slave-p3t5"
I20260810 08:00:16.389002 14870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 08:00:16.486490 14870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 08:00:16.488615 14870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 08:00:16.489234 14870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 08:00:16.498798 14870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 08:00:16.511300 14870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 08:00:16.511677 14870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 08:00:16.512022 14870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 08:00:16.512257 14870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 08:00:16.889191 14870 rpc_server.cc:307] RPC server started. Bound to: 127.14.57.193:43601
I20260810 08:00:16.889338 15038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.57.193:43601 every 8 connection(s)
I20260810 08:00:16.893015 14870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/data/info.pb
I20260810 08:00:16.893685 14567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzwAODw/build/tsan/bin/kudu as pid 14870
I20260810 08:00:16.894297 14567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal/instance
I20260810 08:00:16.909886 14567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzwAODw/build/tsan/bin/kudu
/tmp/dist-test-taskzwAODw/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.57.194:0
--local_ip_for_outbound_sockets=127.14.57.194
--webserver_interface=127.14.57.194
--webserver_port=0
--tserver_master_addrs=127.14.57.254:43757,127.14.57.253:40439,127.14.57.252:33819
--builtin_ntp_servers=127.14.57.212:34009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260810 08:00:16.964217 15046 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e8cadb26fe0428a964f66ef8172f3be)
I20260810 08:00:16.964941 15046 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:16.974804 14705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99f2d868d76c440f8443b33e1cd19f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38a7164890624423a7427871aa2b10e0" is_pre_election: true
I20260810 08:00:16.976076 14791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99f2d868d76c440f8443b33e1cd19f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e8cadb26fe0428a964f66ef8172f3be" is_pre_election: true
I20260810 08:00:16.978186 14597 leader_election.cc:302] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99f2d868d76c440f8443b33e1cd19f58; no voters: 1e8cadb26fe0428a964f66ef8172f3be, 38a7164890624423a7427871aa2b10e0
I20260810 08:00:16.986518 15051 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 08:00:16.989727 15046 leader_election.cc:288] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 38a7164890624423a7427871aa2b10e0 (127.14.57.253:40439), 1e8cadb26fe0428a964f66ef8172f3be (127.14.57.252:33819)
I20260810 08:00:17.008415 15040 heartbeater.cc:347] Connected to a master server at 127.14.57.253:40439
I20260810 08:00:17.008966 15040 heartbeater.cc:464] Registering TS with master...
I20260810 08:00:17.010257 15040 heartbeater.cc:511] Master 127.14.57.253:40439 requested a full tablet report, sending...
I20260810 08:00:17.014161 14695 ts_manager.cc:194] Registered new tserver with Master: a91b27762182454baada71e91ceab224 (127.14.57.193:43601)
I20260810 08:00:17.020700 15039 heartbeater.cc:347] Connected to a master server at 127.14.57.252:33819
I20260810 08:00:17.021099 15039 heartbeater.cc:464] Registering TS with master...
I20260810 08:00:17.022068 15039 heartbeater.cc:511] Master 127.14.57.252:33819 requested a full tablet report, sending...
I20260810 08:00:17.024876 14781 ts_manager.cc:194] Registered new tserver with Master: a91b27762182454baada71e91ceab224 (127.14.57.193:43601)
I20260810 08:00:17.027979 14781 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.57.193:41659
I20260810 08:00:17.030092 15043 heartbeater.cc:347] Connected to a master server at 127.14.57.254:43757
I20260810 08:00:17.030449 15043 heartbeater.cc:464] Registering TS with master...
I20260810 08:00:17.031467 15043 heartbeater.cc:511] Master 127.14.57.254:43757 requested a full tablet report, sending...
I20260810 08:00:17.034906 14612 ts_manager.cc:194] Registered new tserver with Master: a91b27762182454baada71e91ceab224 (127.14.57.193:43601)
W20260810 08:00:17.356513 15050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 08:00:17.357117 15050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 08:00:17.357807 15050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 08:00:17.391068 15050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 08:00:17.392143 15050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.57.194
I20260810 08:00:17.430522 15050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.57.212:34009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.57.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.57.194
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.14.57.254:43757,127.14.57.253:40439,127.14.57.252:33819
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.57.194
--log_dir=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 07:02:49 UTC on ccycloud.quasar-ojudvd.root.comops.site
build id 4577116
TSAN enabled
I20260810 08:00:17.431877 15050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 08:00:17.434136 15050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 08:00:17.458693 15061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 08:00:17.538864 15039 heartbeater.cc:503] Master 127.14.57.252:33819 was elected leader, sending a full tablet report...
I20260810 08:00:17.705137 15067 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e8cadb26fe0428a964f66ef8172f3be)
I20260810 08:00:17.705583 15067 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:17.707342 15067 leader_election.cc:288] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99f2d868d76c440f8443b33e1cd19f58 (127.14.57.254:43757), 1e8cadb26fe0428a964f66ef8172f3be (127.14.57.252:33819)
I20260810 08:00:17.708539 14791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "38a7164890624423a7427871aa2b10e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e8cadb26fe0428a964f66ef8172f3be" is_pre_election: true
I20260810 08:00:17.708747 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "38a7164890624423a7427871aa2b10e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99f2d868d76c440f8443b33e1cd19f58" is_pre_election: true
I20260810 08:00:17.711275 14680 leader_election.cc:302] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38a7164890624423a7427871aa2b10e0; no voters: 1e8cadb26fe0428a964f66ef8172f3be, 99f2d868d76c440f8443b33e1cd19f58
I20260810 08:00:17.712386 15067 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 08:00:18.054744 15068 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e8cadb26fe0428a964f66ef8172f3be)
I20260810 08:00:18.055291 15068 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:18.057098 15068 leader_election.cc:288] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 38a7164890624423a7427871aa2b10e0 (127.14.57.253:40439), 1e8cadb26fe0428a964f66ef8172f3be (127.14.57.252:33819)
I20260810 08:00:18.058385 14791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99f2d868d76c440f8443b33e1cd19f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e8cadb26fe0428a964f66ef8172f3be" is_pre_election: true
I20260810 08:00:18.058383 14705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99f2d868d76c440f8443b33e1cd19f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38a7164890624423a7427871aa2b10e0" is_pre_election: true
I20260810 08:00:18.060232 14598 leader_election.cc:302] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99f2d868d76c440f8443b33e1cd19f58; no voters: 1e8cadb26fe0428a964f66ef8172f3be, 38a7164890624423a7427871aa2b10e0
I20260810 08:00:18.060945 15068 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 08:00:18.593621 15069 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e8cadb26fe0428a964f66ef8172f3be)
I20260810 08:00:18.594144 15069 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:18.595783 15069 leader_election.cc:288] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 38a7164890624423a7427871aa2b10e0 (127.14.57.253:40439), 1e8cadb26fe0428a964f66ef8172f3be (127.14.57.252:33819)
I20260810 08:00:18.596844 14791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99f2d868d76c440f8443b33e1cd19f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e8cadb26fe0428a964f66ef8172f3be" is_pre_election: true
I20260810 08:00:18.596804 14705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99f2d868d76c440f8443b33e1cd19f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38a7164890624423a7427871aa2b10e0" is_pre_election: true
I20260810 08:00:18.598369 14598 leader_election.cc:302] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99f2d868d76c440f8443b33e1cd19f58; no voters: 1e8cadb26fe0428a964f66ef8172f3be, 38a7164890624423a7427871aa2b10e0
I20260810 08:00:18.599123 15069 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 08:00:18.632428 15070 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e8cadb26fe0428a964f66ef8172f3be)
I20260810 08:00:18.632863 15070 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:18.634450 15070 leader_election.cc:288] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99f2d868d76c440f8443b33e1cd19f58 (127.14.57.254:43757), 1e8cadb26fe0428a964f66ef8172f3be (127.14.57.252:33819)
I20260810 08:00:18.635422 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "38a7164890624423a7427871aa2b10e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99f2d868d76c440f8443b33e1cd19f58" is_pre_election: true
I20260810 08:00:18.635560 14791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "38a7164890624423a7427871aa2b10e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e8cadb26fe0428a964f66ef8172f3be" is_pre_election: true
I20260810 08:00:18.637012 14681 leader_election.cc:302] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38a7164890624423a7427871aa2b10e0; no voters: 1e8cadb26fe0428a964f66ef8172f3be, 99f2d868d76c440f8443b33e1cd19f58
I20260810 08:00:18.637825 15070 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 08:00:17.459107 15062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 08:00:17.459987 15050 server_base.cc:1034] running on GCE node
W20260810 08:00:17.460623 15064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 08:00:18.719213 15050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 08:00:18.721680 15050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 08:00:18.723052 15050 hybrid_clock.cc:648] HybridClock initialized: now 1786348818723024 us; error 52 us; skew 500 ppm
I20260810 08:00:18.724014 15050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 08:00:18.730768 15050 webserver.cc:460] Webserver started at http://127.14.57.194:42187/ using document root <none> and password file <none>
I20260810 08:00:18.732067 15050 fs_manager.cc:359] Metadata directory not provided
I20260810 08:00:18.732271 15050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 08:00:18.732679 15050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 08:00:18.737603 15050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/data/instance:
uuid: "9d7219686f304f62a54acc7497cbbd07"
format_stamp: "Formatted at 2026-08-10 08:00:18 on dist-test-slave-p3t5"
I20260810 08:00:18.738862 15050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/instance:
uuid: "9d7219686f304f62a54acc7497cbbd07"
format_stamp: "Formatted at 2026-08-10 08:00:18 on dist-test-slave-p3t5"
I20260810 08:00:18.747614 15050 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.004s
I20260810 08:00:18.755376 15075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 08:00:18.756953 15050 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260810 08:00:18.757400 15050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal
uuid: "9d7219686f304f62a54acc7497cbbd07"
format_stamp: "Formatted at 2026-08-10 08:00:18 on dist-test-slave-p3t5"
I20260810 08:00:18.757967 15050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 08:00:18.821061 15050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 08:00:18.822930 15050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 08:00:18.823513 15050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 08:00:18.826653 15050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 08:00:18.831729 15050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 08:00:18.831976 15050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 08:00:18.832223 15050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 08:00:18.832374 15050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 08:00:19.002347 15050 rpc_server.cc:307] RPC server started. Bound to: 127.14.57.194:36265
I20260810 08:00:19.002449 15198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.57.194:36265 every 8 connection(s)
I20260810 08:00:19.005299 15050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/data/info.pb
I20260810 08:00:19.008138 14567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzwAODw/build/tsan/bin/kudu as pid 15050
I20260810 08:00:19.008857 14567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/instance
I20260810 08:00:19.043141 14567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzwAODw/build/tsan/bin/kudu
/tmp/dist-test-taskzwAODw/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.57.195:0
--local_ip_for_outbound_sockets=127.14.57.195
--webserver_interface=127.14.57.195
--webserver_port=0
--tserver_master_addrs=127.14.57.254:43757,127.14.57.253:40439,127.14.57.252:33819
--builtin_ntp_servers=127.14.57.212:34009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260810 08:00:19.065912 15199 heartbeater.cc:347] Connected to a master server at 127.14.57.252:33819
I20260810 08:00:19.066416 15199 heartbeater.cc:464] Registering TS with master...
I20260810 08:00:19.067734 15199 heartbeater.cc:511] Master 127.14.57.252:33819 requested a full tablet report, sending...
I20260810 08:00:19.068535 15200 heartbeater.cc:347] Connected to a master server at 127.14.57.253:40439
I20260810 08:00:19.068931 15200 heartbeater.cc:464] Registering TS with master...
I20260810 08:00:19.069860 15200 heartbeater.cc:511] Master 127.14.57.253:40439 requested a full tablet report, sending...
I20260810 08:00:19.070690 14781 ts_manager.cc:194] Registered new tserver with Master: 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:19.072659 14781 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.57.194:59021
I20260810 08:00:19.073199 14695 ts_manager.cc:194] Registered new tserver with Master: 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:19.077759 15202 heartbeater.cc:347] Connected to a master server at 127.14.57.254:43757
I20260810 08:00:19.078179 15202 heartbeater.cc:464] Registering TS with master...
I20260810 08:00:19.078908 15202 heartbeater.cc:511] Master 127.14.57.254:43757 requested a full tablet report, sending...
I20260810 08:00:19.081595 14612 ts_manager.cc:194] Registered new tserver with Master: 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
W20260810 08:00:19.381299 15210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 08:00:19.382058 15210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 08:00:19.382695 15210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 08:00:19.416101 15210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 08:00:19.417214 15210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.57.195
I20260810 08:00:19.453589 15210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.57.212:34009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.57.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.57.195
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.14.57.254:43757,127.14.57.253:40439,127.14.57.252:33819
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.57.195
--log_dir=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 07:02:49 UTC on ccycloud.quasar-ojudvd.root.comops.site
build id 4577116
TSAN enabled
I20260810 08:00:19.455003 15210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 08:00:19.456911 15210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 08:00:19.577250 15199 heartbeater.cc:503] Master 127.14.57.252:33819 was elected leader, sending a full tablet report...
I20260810 08:00:20.401280 15224 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e8cadb26fe0428a964f66ef8172f3be)
I20260810 08:00:20.402006 15224 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:20.403836 15224 leader_election.cc:288] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99f2d868d76c440f8443b33e1cd19f58 (127.14.57.254:43757), 1e8cadb26fe0428a964f66ef8172f3be (127.14.57.252:33819)
I20260810 08:00:20.405107 14791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "38a7164890624423a7427871aa2b10e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e8cadb26fe0428a964f66ef8172f3be" is_pre_election: true
I20260810 08:00:20.405436 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "38a7164890624423a7427871aa2b10e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "99f2d868d76c440f8443b33e1cd19f58" is_pre_election: true
I20260810 08:00:20.407629 14680 leader_election.cc:302] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38a7164890624423a7427871aa2b10e0; no voters: 1e8cadb26fe0428a964f66ef8172f3be, 99f2d868d76c440f8443b33e1cd19f58
I20260810 08:00:20.408777 15224 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 08:00:19.478551 15220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 08:00:19.478576 15217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 08:00:19.481353 15210 server_base.cc:1034] running on GCE node
W20260810 08:00:19.478513 15218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 08:00:20.708720 15210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 08:00:20.711822 15210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 08:00:20.713296 15210 hybrid_clock.cc:648] HybridClock initialized: now 1786348820713249 us; error 68 us; skew 500 ppm
I20260810 08:00:20.714202 15210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 08:00:20.721598 15210 webserver.cc:460] Webserver started at http://127.14.57.195:33163/ using document root <none> and password file <none>
I20260810 08:00:20.722815 15210 fs_manager.cc:359] Metadata directory not provided
I20260810 08:00:20.723070 15210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 08:00:20.723605 15210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 08:00:20.728926 15210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/data/instance:
uuid: "3fc5e568f3724f158b616c303efb9eae"
format_stamp: "Formatted at 2026-08-10 08:00:20 on dist-test-slave-p3t5"
I20260810 08:00:20.730146 15210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal/instance:
uuid: "3fc5e568f3724f158b616c303efb9eae"
format_stamp: "Formatted at 2026-08-10 08:00:20 on dist-test-slave-p3t5"
I20260810 08:00:20.737735 15210 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260810 08:00:20.743928 15229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 08:00:20.745329 15210 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20260810 08:00:20.745752 15210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal
uuid: "3fc5e568f3724f158b616c303efb9eae"
format_stamp: "Formatted at 2026-08-10 08:00:20 on dist-test-slave-p3t5"
I20260810 08:00:20.746199 15210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 08:00:20.804884 15210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 08:00:20.806604 15210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 08:00:20.807142 15210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 08:00:20.810045 15210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 08:00:20.814838 15210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 08:00:20.815084 15210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 08:00:20.815457 15210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 08:00:20.815658 15210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 08:00:20.965183 15330 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e8cadb26fe0428a964f66ef8172f3be)
I20260810 08:00:20.965853 15330 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:20.968510 15330 leader_election.cc:288] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 38a7164890624423a7427871aa2b10e0 (127.14.57.253:40439), 1e8cadb26fe0428a964f66ef8172f3be (127.14.57.252:33819)
I20260810 08:00:20.969718 14705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99f2d868d76c440f8443b33e1cd19f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38a7164890624423a7427871aa2b10e0" is_pre_election: true
I20260810 08:00:20.972234 14791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99f2d868d76c440f8443b33e1cd19f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e8cadb26fe0428a964f66ef8172f3be" is_pre_election: true
I20260810 08:00:20.974202 14598 leader_election.cc:302] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99f2d868d76c440f8443b33e1cd19f58; no voters: 1e8cadb26fe0428a964f66ef8172f3be, 38a7164890624423a7427871aa2b10e0
I20260810 08:00:20.975780 15330 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 08:00:20.997944 15210 rpc_server.cc:307] RPC server started. Bound to: 127.14.57.195:37729
I20260810 08:00:20.998028 15353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.57.195:37729 every 8 connection(s)
I20260810 08:00:21.000882 15210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/data/info.pb
I20260810 08:00:21.010779 14567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzwAODw/build/tsan/bin/kudu as pid 15210
I20260810 08:00:21.011425 14567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal/instance
I20260810 08:00:21.062567 15354 heartbeater.cc:347] Connected to a master server at 127.14.57.252:33819
I20260810 08:00:21.063257 15354 heartbeater.cc:464] Registering TS with master...
I20260810 08:00:21.064899 15354 heartbeater.cc:511] Master 127.14.57.252:33819 requested a full tablet report, sending...
I20260810 08:00:21.069145 14781 ts_manager.cc:194] Registered new tserver with Master: 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:21.072491 14781 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.57.195:50291
I20260810 08:00:21.081562 15355 heartbeater.cc:347] Connected to a master server at 127.14.57.253:40439
I20260810 08:00:21.082006 15355 heartbeater.cc:464] Registering TS with master...
I20260810 08:00:21.082758 15355 heartbeater.cc:511] Master 127.14.57.253:40439 requested a full tablet report, sending...
I20260810 08:00:21.085669 14695 ts_manager.cc:194] Registered new tserver with Master: 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:21.092150 15356 heartbeater.cc:347] Connected to a master server at 127.14.57.254:43757
I20260810 08:00:21.092519 15356 heartbeater.cc:464] Registering TS with master...
I20260810 08:00:21.093197 15356 heartbeater.cc:511] Master 127.14.57.254:43757 requested a full tablet report, sending...
I20260810 08:00:21.095772 14611 ts_manager.cc:194] Registered new tserver with Master: 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:21.105252 14567 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 08:00:21.166849 14781 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50988:
name: "default.Table_8ac532f4969e45a197db311de2a7b795"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:21.170692 14780 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50988:
name: "default.Table_0e30c621046242609c9f6fc73b16d0e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:21.171339 14779 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50988:
name: "default.Table_5d982fdf0c924d0ebecb9de16fdebd15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:21.172482 14780 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e30c621046242609c9f6fc73b16d0e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 08:00:21.173734 14779 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d982fdf0c924d0ebecb9de16fdebd15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:21.174336 14778 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50988:
name: "default.Table_7f3a4e19ded64b3783a00b3d4c6839c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:21.176302 14778 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f3a4e19ded64b3783a00b3d4c6839c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 08:00:21.170717 14781 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ac532f4969e45a197db311de2a7b795 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:21.245783 14567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzwAODw/build/tsan/bin/kudu with pid 14735
W20260810 08:00:21.290875 14679 connection.cc:535] server connection from 127.0.0.1:60280 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 08:00:21.291605 14597 connection.cc:535] server connection from 127.0.0.1:54362 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 08:00:21.581316 15354 heartbeater.cc:503] Master 127.14.57.252:33819 was elected leader, sending a full tablet report...
W20260810 08:00:21.585469 15354 heartbeater.cc:644] Failed to heartbeat to 127.14.57.252:33819 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.57.252:33819: connect: Connection refused (error 111)
W20260810 08:00:21.586335 15039 heartbeater.cc:644] Failed to heartbeat to 127.14.57.252:33819 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.57.252:33819: connect: Connection refused (error 111)
W20260810 08:00:21.598536 15199 heartbeater.cc:644] Failed to heartbeat to 127.14.57.252:33819 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.57.252:33819: connect: Connection refused (error 111)
I20260810 08:00:21.740161 15402 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e8cadb26fe0428a964f66ef8172f3be)
I20260810 08:00:21.740646 15402 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:21.742305 15402 leader_election.cc:288] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 38a7164890624423a7427871aa2b10e0 (127.14.57.253:40439), 1e8cadb26fe0428a964f66ef8172f3be (127.14.57.252:33819)
I20260810 08:00:21.743170 14705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99f2d868d76c440f8443b33e1cd19f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "38a7164890624423a7427871aa2b10e0" is_pre_election: true
W20260810 08:00:21.745276 14598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.57.252:33819: connect: Connection refused (error 111) [suppressed 13 similar messages]
I20260810 08:00:21.747233 15404 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e8cadb26fe0428a964f66ef8172f3be)
I20260810 08:00:21.748150 15404 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:21.750931 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "38a7164890624423a7427871aa2b10e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "99f2d868d76c440f8443b33e1cd19f58" is_pre_election: true
I20260810 08:00:21.751561 14623 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38a7164890624423a7427871aa2b10e0 in term 1.
I20260810 08:00:21.751525 15404 leader_election.cc:288] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99f2d868d76c440f8443b33e1cd19f58 (127.14.57.254:43757), 1e8cadb26fe0428a964f66ef8172f3be (127.14.57.252:33819)
W20260810 08:00:21.752673 14598 leader_election.cc:334] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1e8cadb26fe0428a964f66ef8172f3be (127.14.57.252:33819): Network error: Client connection negotiation failed: client connection to 127.14.57.252:33819: connect: Connection refused (error 111)
I20260810 08:00:21.752769 14680 leader_election.cc:302] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38a7164890624423a7427871aa2b10e0, 99f2d868d76c440f8443b33e1cd19f58; no voters:
I20260810 08:00:21.753275 14598 leader_election.cc:302] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99f2d868d76c440f8443b33e1cd19f58; no voters: 1e8cadb26fe0428a964f66ef8172f3be, 38a7164890624423a7427871aa2b10e0
I20260810 08:00:21.754581 15402 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 08:00:21.754725 15404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 08:00:21.755079 15404 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e8cadb26fe0428a964f66ef8172f3be)
I20260810 08:00:21.755475 15404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 1 FOLLOWER]: Advancing to term 2
W20260810 08:00:21.757268 14681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.57.252:33819: connect: Connection refused (error 111) [suppressed 6 similar messages]
W20260810 08:00:21.762317 14681 leader_election.cc:334] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1e8cadb26fe0428a964f66ef8172f3be (127.14.57.252:33819): Network error: Client connection negotiation failed: client connection to 127.14.57.252:33819: connect: Connection refused (error 111)
I20260810 08:00:21.763885 15404 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:21.765456 15404 leader_election.cc:288] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [CANDIDATE]: Term 2 election: Requested vote from peers 99f2d868d76c440f8443b33e1cd19f58 (127.14.57.254:43757), 1e8cadb26fe0428a964f66ef8172f3be (127.14.57.252:33819)
I20260810 08:00:21.766608 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "38a7164890624423a7427871aa2b10e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "99f2d868d76c440f8443b33e1cd19f58"
I20260810 08:00:21.767472 14623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 1 FOLLOWER]: Advancing to term 2
W20260810 08:00:21.769289 14681 leader_election.cc:334] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1e8cadb26fe0428a964f66ef8172f3be (127.14.57.252:33819): Network error: Client connection negotiation failed: client connection to 127.14.57.252:33819: connect: Connection refused (error 111)
I20260810 08:00:21.779598 14623 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38a7164890624423a7427871aa2b10e0 in term 2.
I20260810 08:00:21.781080 14680 leader_election.cc:302] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38a7164890624423a7427871aa2b10e0, 99f2d868d76c440f8443b33e1cd19f58; no voters: 1e8cadb26fe0428a964f66ef8172f3be
I20260810 08:00:21.781838 15404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 08:00:21.783089 15404 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 2 LEADER]: Becoming Leader. State: Replica: 38a7164890624423a7427871aa2b10e0, State: Running, Role: LEADER
I20260810 08:00:21.784024 15404 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:21.787307 15408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38a7164890624423a7427871aa2b10e0. Latest consensus state: current_term: 2 leader_uuid: "38a7164890624423a7427871aa2b10e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } } }
I20260810 08:00:21.787940 15408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [sys.catalog]: This master's current role is: LEADER
W20260810 08:00:22.249207 14681 consensus_peers.cc:487] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 -> Peer 1e8cadb26fe0428a964f66ef8172f3be (127.14.57.252:33819): Couldn't send request to peer 1e8cadb26fe0428a964f66ef8172f3be. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.252:33819: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 08:00:22.351552 15414 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:22.351955 15414 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:22.353523 15414 leader_election.cc:288] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 38a7164890624423a7427871aa2b10e0 (127.14.57.253:40439), 1e8cadb26fe0428a964f66ef8172f3be (127.14.57.252:33819)
I20260810 08:00:22.355033 14705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99f2d868d76c440f8443b33e1cd19f58" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "38a7164890624423a7427871aa2b10e0" is_pre_election: true
W20260810 08:00:22.359226 14598 leader_election.cc:334] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1e8cadb26fe0428a964f66ef8172f3be (127.14.57.252:33819): Network error: Client connection negotiation failed: client connection to 127.14.57.252:33819: connect: Connection refused (error 111)
I20260810 08:00:22.359606 14598 leader_election.cc:302] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99f2d868d76c440f8443b33e1cd19f58; no voters: 1e8cadb26fe0428a964f66ef8172f3be, 38a7164890624423a7427871aa2b10e0
I20260810 08:00:22.360244 15414 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 08:00:22.409636 14623 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 2 FOLLOWER]: Refusing update from remote peer 38a7164890624423a7427871aa2b10e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20260810 08:00:22.411052 15408 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 08:00:22.415897 15414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38a7164890624423a7427871aa2b10e0. Latest consensus state: current_term: 2 leader_uuid: "38a7164890624423a7427871aa2b10e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } } }
I20260810 08:00:22.416725 15414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 08:00:22.434674 15408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "38a7164890624423a7427871aa2b10e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } } }
I20260810 08:00:22.435209 15408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [sys.catalog]: This master's current role is: LEADER
I20260810 08:00:22.437395 15409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 08:00:22.437786 15414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "38a7164890624423a7427871aa2b10e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } } }
I20260810 08:00:22.438509 15414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 08:00:22.446089 15409 catalog_manager.cc:669] Loaded metadata for table default.Table_7f3a4e19ded64b3783a00b3d4c6839c3 [id=6c6afcf578534997878b98feb9574e2d]
I20260810 08:00:22.447491 15409 catalog_manager.cc:669] Loaded metadata for table default.Table_8ac532f4969e45a197db311de2a7b795 [id=7893824f2c374508b7b3c4635ebea5b6]
I20260810 08:00:22.448872 15409 catalog_manager.cc:669] Loaded metadata for table default.Table_5d982fdf0c924d0ebecb9de16fdebd15 [id=cd26deac07054fd6b8667aae1f1e272b]
I20260810 08:00:22.450234 15409 catalog_manager.cc:669] Loaded metadata for table default.Table_0e30c621046242609c9f6fc73b16d0e7 [id=e9608d66a9964bc6aafdf84f574fba02]
I20260810 08:00:22.454550 15409 tablet_loader.cc:96] loaded metadata for tablet 06e3130f1b2744d1907390c4f2ce8ce1 (table default.Table_8ac532f4969e45a197db311de2a7b795 [id=7893824f2c374508b7b3c4635ebea5b6])
I20260810 08:00:22.455446 15409 tablet_loader.cc:96] loaded metadata for tablet 3cdca8a5b59a449fb905065b997474e3 (table default.Table_5d982fdf0c924d0ebecb9de16fdebd15 [id=cd26deac07054fd6b8667aae1f1e272b])
I20260810 08:00:22.456386 15409 tablet_loader.cc:96] loaded metadata for tablet b370eb11d5ab4a6e8a253cd0c36d3ca4 (table default.Table_7f3a4e19ded64b3783a00b3d4c6839c3 [id=6c6afcf578534997878b98feb9574e2d])
I20260810 08:00:22.457216 15409 tablet_loader.cc:96] loaded metadata for tablet e0206444ca194f8f98e9d71aa858283d (table default.Table_0e30c621046242609c9f6fc73b16d0e7 [id=e9608d66a9964bc6aafdf84f574fba02])
I20260810 08:00:22.458159 15409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 08:00:22.460842 15409 catalog_manager.cc:1259] Loaded cluster ID: eddaf5b0e44343438f86652258a1a0e1
I20260810 08:00:22.461073 15409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 08:00:22.464419 15409 catalog_manager.cc:1504] Loading token signing keys...
I20260810 08:00:22.467273 15409 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Loaded TSK: 0
I20260810 08:00:22.468683 15409 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 08:00:22.586622 14966 tablet_service.cc:1467] Processing CreateTablet for tablet e0206444ca194f8f98e9d71aa858283d (DEFAULT_TABLE table=default.Table_0e30c621046242609c9f6fc73b16d0e7 [id=e9608d66a9964bc6aafdf84f574fba02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:22.588558 14966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0206444ca194f8f98e9d71aa858283d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:22.590139 14965 tablet_service.cc:1467] Processing CreateTablet for tablet 3cdca8a5b59a449fb905065b997474e3 (DEFAULT_TABLE table=default.Table_5d982fdf0c924d0ebecb9de16fdebd15 [id=cd26deac07054fd6b8667aae1f1e272b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:22.603551 14965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cdca8a5b59a449fb905065b997474e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:22.603065 15280 tablet_service.cc:1467] Processing CreateTablet for tablet 06e3130f1b2744d1907390c4f2ce8ce1 (DEFAULT_TABLE table=default.Table_8ac532f4969e45a197db311de2a7b795 [id=7893824f2c374508b7b3c4635ebea5b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:22.603296 15279 tablet_service.cc:1467] Processing CreateTablet for tablet e0206444ca194f8f98e9d71aa858283d (DEFAULT_TABLE table=default.Table_0e30c621046242609c9f6fc73b16d0e7 [id=e9608d66a9964bc6aafdf84f574fba02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:22.603441 15126 tablet_service.cc:1467] Processing CreateTablet for tablet 06e3130f1b2744d1907390c4f2ce8ce1 (DEFAULT_TABLE table=default.Table_8ac532f4969e45a197db311de2a7b795 [id=7893824f2c374508b7b3c4635ebea5b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:22.604480 15277 tablet_service.cc:1467] Processing CreateTablet for tablet b370eb11d5ab4a6e8a253cd0c36d3ca4 (DEFAULT_TABLE table=default.Table_7f3a4e19ded64b3783a00b3d4c6839c3 [id=6c6afcf578534997878b98feb9574e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:22.591588 14964 tablet_service.cc:1467] Processing CreateTablet for tablet b370eb11d5ab4a6e8a253cd0c36d3ca4 (DEFAULT_TABLE table=default.Table_7f3a4e19ded64b3783a00b3d4c6839c3 [id=6c6afcf578534997878b98feb9574e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:22.605049 15280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06e3130f1b2744d1907390c4f2ce8ce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:22.605245 15126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06e3130f1b2744d1907390c4f2ce8ce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:22.605754 14964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b370eb11d5ab4a6e8a253cd0c36d3ca4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:22.605684 15278 tablet_service.cc:1467] Processing CreateTablet for tablet 3cdca8a5b59a449fb905065b997474e3 (DEFAULT_TABLE table=default.Table_5d982fdf0c924d0ebecb9de16fdebd15 [id=cd26deac07054fd6b8667aae1f1e272b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:22.606876 15278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cdca8a5b59a449fb905065b997474e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:22.608108 15122 tablet_service.cc:1467] Processing CreateTablet for tablet b370eb11d5ab4a6e8a253cd0c36d3ca4 (DEFAULT_TABLE table=default.Table_7f3a4e19ded64b3783a00b3d4c6839c3 [id=6c6afcf578534997878b98feb9574e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:22.585235 14967 tablet_service.cc:1467] Processing CreateTablet for tablet 06e3130f1b2744d1907390c4f2ce8ce1 (DEFAULT_TABLE table=default.Table_8ac532f4969e45a197db311de2a7b795 [id=7893824f2c374508b7b3c4635ebea5b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:22.608136 15123 tablet_service.cc:1467] Processing CreateTablet for tablet 3cdca8a5b59a449fb905065b997474e3 (DEFAULT_TABLE table=default.Table_5d982fdf0c924d0ebecb9de16fdebd15 [id=cd26deac07054fd6b8667aae1f1e272b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:22.610695 15122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b370eb11d5ab4a6e8a253cd0c36d3ca4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:22.610594 15125 tablet_service.cc:1467] Processing CreateTablet for tablet e0206444ca194f8f98e9d71aa858283d (DEFAULT_TABLE table=default.Table_0e30c621046242609c9f6fc73b16d0e7 [id=e9608d66a9964bc6aafdf84f574fba02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:22.611725 15125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0206444ca194f8f98e9d71aa858283d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:22.613911 15123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cdca8a5b59a449fb905065b997474e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:22.617075 15277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b370eb11d5ab4a6e8a253cd0c36d3ca4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:22.617590 14967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06e3130f1b2744d1907390c4f2ce8ce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:22.622944 15279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0206444ca194f8f98e9d71aa858283d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:22.700580 15437 tablet_bootstrap.cc:492] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:22.725420 15438 tablet_bootstrap.cc:492] T 3cdca8a5b59a449fb905065b997474e3 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:22.725966 15439 tablet_bootstrap.cc:492] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:22.732184 15437 tablet_bootstrap.cc:654] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:22.732646 15438 tablet_bootstrap.cc:654] T 3cdca8a5b59a449fb905065b997474e3 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:22.734748 15439 tablet_bootstrap.cc:654] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:22.735210 15438 log.cc:824] T 3cdca8a5b59a449fb905065b997474e3 P 9d7219686f304f62a54acc7497cbbd07: Log is configured to *not* fsync() on all Append() calls
I20260810 08:00:22.735394 15437 log.cc:824] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P a91b27762182454baada71e91ceab224: Log is configured to *not* fsync() on all Append() calls
I20260810 08:00:22.737375 15439 log.cc:824] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P 3fc5e568f3724f158b616c303efb9eae: Log is configured to *not* fsync() on all Append() calls
I20260810 08:00:22.744344 15437 tablet_bootstrap.cc:492] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:22.744575 15438 tablet_bootstrap.cc:492] T 3cdca8a5b59a449fb905065b997474e3 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:22.744720 15439 tablet_bootstrap.cc:492] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:22.744939 15437 ts_tablet_manager.cc:1397] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.045s user 0.008s sys 0.012s
I20260810 08:00:22.745060 15438 ts_tablet_manager.cc:1397] T 3cdca8a5b59a449fb905065b997474e3 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.008s
I20260810 08:00:22.745296 15439 ts_tablet_manager.cc:1397] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.004s
I20260810 08:00:22.778224 15437 raft_consensus.cc:348] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:22.778232 15438 raft_consensus.cc:348] T 3cdca8a5b59a449fb905065b997474e3 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:22.779237 15437 raft_consensus.cc:374] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:22.779230 15438 raft_consensus.cc:374] T 3cdca8a5b59a449fb905065b997474e3 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:22.779559 15438 raft_consensus.cc:729] T 3cdca8a5b59a449fb905065b997474e3 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:22.779559 15437 raft_consensus.cc:729] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:22.780588 15438 consensus_queue.cc:260] T 3cdca8a5b59a449fb905065b997474e3 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:22.780579 15437 consensus_queue.cc:260] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:22.781852 15439 raft_consensus.cc:348] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:22.782987 15439 raft_consensus.cc:374] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:22.783331 15439 raft_consensus.cc:729] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:22.784950 15200 heartbeater.cc:503] Master 127.14.57.253:40439 was elected leader, sending a full tablet report...
I20260810 08:00:22.785811 15438 ts_tablet_manager.cc:1428] T 3cdca8a5b59a449fb905065b997474e3 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.040s user 0.034s sys 0.008s
I20260810 08:00:22.787189 15438 tablet_bootstrap.cc:492] T 06e3130f1b2744d1907390c4f2ce8ce1 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
W20260810 08:00:22.788614 15204 tablet.cc:2369] T 3cdca8a5b59a449fb905065b997474e3 P 9d7219686f304f62a54acc7497cbbd07: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 08:00:22.796387 15438 tablet_bootstrap.cc:654] T 06e3130f1b2744d1907390c4f2ce8ce1 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:22.807010 15439 consensus_queue.cc:260] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:22.813522 15439 ts_tablet_manager.cc:1428] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.068s user 0.032s sys 0.012s
I20260810 08:00:22.817406 15040 heartbeater.cc:503] Master 127.14.57.253:40439 was elected leader, sending a full tablet report...
I20260810 08:00:22.818161 15437 ts_tablet_manager.cc:1428] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.073s user 0.038s sys 0.018s
I20260810 08:00:22.819428 15437 tablet_bootstrap.cc:492] T 06e3130f1b2744d1907390c4f2ce8ce1 P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:22.820129 15355 heartbeater.cc:503] Master 127.14.57.253:40439 was elected leader, sending a full tablet report...
I20260810 08:00:22.825695 15439 tablet_bootstrap.cc:492] T e0206444ca194f8f98e9d71aa858283d P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:22.827745 15437 tablet_bootstrap.cc:654] T 06e3130f1b2744d1907390c4f2ce8ce1 P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:22.833876 15439 tablet_bootstrap.cc:654] T e0206444ca194f8f98e9d71aa858283d P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:22.834656 15438 tablet_bootstrap.cc:492] T 06e3130f1b2744d1907390c4f2ce8ce1 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:22.835141 15438 ts_tablet_manager.cc:1397] T 06e3130f1b2744d1907390c4f2ce8ce1 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.048s user 0.011s sys 0.025s
I20260810 08:00:22.838514 15438 raft_consensus.cc:348] T 06e3130f1b2744d1907390c4f2ce8ce1 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:22.839437 15438 raft_consensus.cc:374] T 06e3130f1b2744d1907390c4f2ce8ce1 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:22.839744 15438 raft_consensus.cc:729] T 06e3130f1b2744d1907390c4f2ce8ce1 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:22.840543 15438 consensus_queue.cc:260] T 06e3130f1b2744d1907390c4f2ce8ce1 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:22.843010 15438 ts_tablet_manager.cc:1428] T 06e3130f1b2744d1907390c4f2ce8ce1 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260810 08:00:22.847069 15438 tablet_bootstrap.cc:492] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:22.853854 15438 tablet_bootstrap.cc:654] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:22.866264 15438 tablet_bootstrap.cc:492] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:22.866837 15438 ts_tablet_manager.cc:1397] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.008s
I20260810 08:00:22.869874 15438 raft_consensus.cc:348] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:22.870697 15438 raft_consensus.cc:374] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:22.870994 15438 raft_consensus.cc:729] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:22.874440 15438 consensus_queue.cc:260] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:22.884773 15438 ts_tablet_manager.cc:1428] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.018s user 0.014s sys 0.000s
I20260810 08:00:22.891992 15438 tablet_bootstrap.cc:492] T e0206444ca194f8f98e9d71aa858283d P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:22.897934 15438 tablet_bootstrap.cc:654] T e0206444ca194f8f98e9d71aa858283d P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:22.904371 15438 tablet_bootstrap.cc:492] T e0206444ca194f8f98e9d71aa858283d P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:22.904855 15438 ts_tablet_manager.cc:1397] T e0206444ca194f8f98e9d71aa858283d P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260810 08:00:22.907444 15438 raft_consensus.cc:348] T e0206444ca194f8f98e9d71aa858283d P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:22.908201 15438 raft_consensus.cc:374] T e0206444ca194f8f98e9d71aa858283d P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:22.908489 15438 raft_consensus.cc:729] T e0206444ca194f8f98e9d71aa858283d P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:22.909235 15438 consensus_queue.cc:260] T e0206444ca194f8f98e9d71aa858283d P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:22.914247 15438 ts_tablet_manager.cc:1428] T e0206444ca194f8f98e9d71aa858283d P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
W20260810 08:00:21.293861 15367 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.14.57.252:33819)
W20260810 08:00:22.922794 15367 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.CreateTable -> {remote=127.14.57.252:33819, user_credentials={real_user=slave}} blocked reactor thread for 1.63311e+06us
W20260810 08:00:22.589123 14572 debug-util.cc:398] Leaking SignalData structure 0x7b080002a0a0 after lost signal to thread 15367
W20260810 08:00:22.928435 14681 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.57.252:33819, user_credentials={real_user=slave}} blocked reactor thread for 129110us
I20260810 08:00:22.928622 14567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzwAODw/build/tsan/bin/kudu
/tmp/dist-test-taskzwAODw/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.57.252:33819
--webserver_interface=127.14.57.252
--webserver_port=34789
--builtin_ntp_servers=127.14.57.212:34009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.57.254:43757,127.14.57.253:40439,127.14.57.252:33819
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260810 08:00:22.924098 15367 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.14.57.252:33819)
I20260810 08:00:22.945956 15439 tablet_bootstrap.cc:492] T e0206444ca194f8f98e9d71aa858283d P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:22.946488 15439 ts_tablet_manager.cc:1397] T e0206444ca194f8f98e9d71aa858283d P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.121s user 0.044s sys 0.031s
I20260810 08:00:22.946537 15437 tablet_bootstrap.cc:492] T 06e3130f1b2744d1907390c4f2ce8ce1 P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:22.947022 15437 ts_tablet_manager.cc:1397] T 06e3130f1b2744d1907390c4f2ce8ce1 P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.128s user 0.047s sys 0.054s
I20260810 08:00:22.950117 15437 raft_consensus.cc:348] T 06e3130f1b2744d1907390c4f2ce8ce1 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:22.950951 15437 raft_consensus.cc:374] T 06e3130f1b2744d1907390c4f2ce8ce1 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:22.951238 15437 raft_consensus.cc:729] T 06e3130f1b2744d1907390c4f2ce8ce1 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:22.950822 15439 raft_consensus.cc:348] T e0206444ca194f8f98e9d71aa858283d P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:22.951620 15439 raft_consensus.cc:374] T e0206444ca194f8f98e9d71aa858283d P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:22.951915 15439 raft_consensus.cc:729] T e0206444ca194f8f98e9d71aa858283d P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:22.951972 15437 consensus_queue.cc:260] T 06e3130f1b2744d1907390c4f2ce8ce1 P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:22.954684 15437 ts_tablet_manager.cc:1428] T 06e3130f1b2744d1907390c4f2ce8ce1 P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 08:00:22.955646 15437 tablet_bootstrap.cc:492] T 3cdca8a5b59a449fb905065b997474e3 P a91b27762182454baada71e91ceab224: Bootstrap starting.
W20260810 08:00:22.955789 15367 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.14.57.252:33819)
I20260810 08:00:22.957880 15439 consensus_queue.cc:260] T e0206444ca194f8f98e9d71aa858283d P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:22.962364 15437 tablet_bootstrap.cc:654] T 3cdca8a5b59a449fb905065b997474e3 P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:22.969735 15439 ts_tablet_manager.cc:1428] T e0206444ca194f8f98e9d71aa858283d P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.023s user 0.008s sys 0.000s
I20260810 08:00:22.973359 15439 tablet_bootstrap.cc:492] T 06e3130f1b2744d1907390c4f2ce8ce1 P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:22.978941 14693 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_7f3a4e19ded64b3783a00b3d4c6839c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:22.982831 14690 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_8ac532f4969e45a197db311de2a7b795"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:22.983978 14691 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_0e30c621046242609c9f6fc73b16d0e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:22.984730 15439 tablet_bootstrap.cc:654] T 06e3130f1b2744d1907390c4f2ce8ce1 P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:22.982743 14692 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_5d982fdf0c924d0ebecb9de16fdebd15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:22.985514 15047 tablet.cc:2369] T 06e3130f1b2744d1907390c4f2ce8ce1 P a91b27762182454baada71e91ceab224: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 08:00:22.995126 15437 tablet_bootstrap.cc:492] T 3cdca8a5b59a449fb905065b997474e3 P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:22.995609 15437 ts_tablet_manager.cc:1397] T 3cdca8a5b59a449fb905065b997474e3 P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.040s user 0.009s sys 0.003s
I20260810 08:00:22.998219 14689 master_service.cc:946] ReplaceTablet: received request to replace tablet b370eb11d5ab4a6e8a253cd0c36d3ca4 from {username='slave'} at 127.0.0.1:42300
I20260810 08:00:22.998174 15437 raft_consensus.cc:348] T 3cdca8a5b59a449fb905065b997474e3 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:22.998934 15437 raft_consensus.cc:374] T 3cdca8a5b59a449fb905065b997474e3 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:22.999200 15437 raft_consensus.cc:729] T 3cdca8a5b59a449fb905065b997474e3 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:22.999825 15437 consensus_queue.cc:260] T 3cdca8a5b59a449fb905065b997474e3 P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:23.002151 15437 ts_tablet_manager.cc:1428] T 3cdca8a5b59a449fb905065b997474e3 P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260810 08:00:23.003095 15437 tablet_bootstrap.cc:492] T e0206444ca194f8f98e9d71aa858283d P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:23.007193 14688 master_service.cc:946] ReplaceTablet: received request to replace tablet 3cdca8a5b59a449fb905065b997474e3 from {username='slave'} at 127.0.0.1:42300
I20260810 08:00:23.009074 15437 tablet_bootstrap.cc:654] T e0206444ca194f8f98e9d71aa858283d P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
W20260810 08:00:23.011993 14690 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ac532f4969e45a197db311de2a7b795 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 08:00:23.033687 14692 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d982fdf0c924d0ebecb9de16fdebd15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 08:00:23.037432 14693 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f3a4e19ded64b3783a00b3d4c6839c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 08:00:23.027611 14691 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e30c621046242609c9f6fc73b16d0e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 08:00:23.048012 15361 tablet.cc:2369] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P 3fc5e568f3724f158b616c303efb9eae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 08:00:23.060941 15460 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38a7164890624423a7427871aa2b10e0)
I20260810 08:00:23.061453 15460 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:23.067365 15460 leader_election.cc:288] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 38a7164890624423a7427871aa2b10e0 (127.14.57.253:40439), 1e8cadb26fe0428a964f66ef8172f3be (127.14.57.252:33819)
I20260810 08:00:23.078032 15447 raft_consensus.cc:482] T e0206444ca194f8f98e9d71aa858283d P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:23.078820 15447 raft_consensus.cc:504] T e0206444ca194f8f98e9d71aa858283d P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:23.086603 14705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99f2d868d76c440f8443b33e1cd19f58" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "38a7164890624423a7427871aa2b10e0" is_pre_election: true
W20260810 08:00:23.092278 14598 leader_election.cc:334] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1e8cadb26fe0428a964f66ef8172f3be (127.14.57.252:33819): Network error: Client connection negotiation failed: client connection to 127.14.57.252:33819: connect: Connection refused (error 111)
I20260810 08:00:23.104588 15447 leader_election.cc:288] T e0206444ca194f8f98e9d71aa858283d P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:23.114285 14597 leader_election.cc:302] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99f2d868d76c440f8443b33e1cd19f58; no voters: 1e8cadb26fe0428a964f66ef8172f3be, 38a7164890624423a7427871aa2b10e0
I20260810 08:00:23.115166 15460 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 08:00:23.146987 15443 raft_consensus.cc:482] T 3cdca8a5b59a449fb905065b997474e3 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:23.147581 15443 raft_consensus.cc:504] T 3cdca8a5b59a449fb905065b997474e3 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:23.155495 15443 leader_election.cc:288] T 3cdca8a5b59a449fb905065b997474e3 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:23.167065 14694 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_71132b29527746dd98788028dd7c2016"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:23.169118 14694 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_71132b29527746dd98788028dd7c2016 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:23.169480 15439 tablet_bootstrap.cc:492] T 06e3130f1b2744d1907390c4f2ce8ce1 P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:23.170079 15439 ts_tablet_manager.cc:1397] T 06e3130f1b2744d1907390c4f2ce8ce1 P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.197s user 0.010s sys 0.003s
I20260810 08:00:23.170647 14693 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_0f90d62d68f14a6895531b34d93d7d87"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:23.172626 14693 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f90d62d68f14a6895531b34d93d7d87 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:23.172889 15439 raft_consensus.cc:348] T 06e3130f1b2744d1907390c4f2ce8ce1 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:23.173728 15439 raft_consensus.cc:374] T 06e3130f1b2744d1907390c4f2ce8ce1 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:23.174062 15439 raft_consensus.cc:729] T 06e3130f1b2744d1907390c4f2ce8ce1 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:23.174521 14691 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_8db9299a79354b7595c1b2522f07cbec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:23.174962 15439 consensus_queue.cc:260] T 06e3130f1b2744d1907390c4f2ce8ce1 P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
W20260810 08:00:23.176448 14691 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8db9299a79354b7595c1b2522f07cbec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:23.177416 15439 ts_tablet_manager.cc:1428] T 06e3130f1b2744d1907390c4f2ce8ce1 P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.007s user 0.001s sys 0.004s
I20260810 08:00:23.177780 14692 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_58c3137b04464b6d845b2a9a753886eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:23.179569 14692 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_58c3137b04464b6d845b2a9a753886eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:23.185257 15439 tablet_bootstrap.cc:492] T 3cdca8a5b59a449fb905065b997474e3 P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:23.191431 15439 tablet_bootstrap.cc:654] T 3cdca8a5b59a449fb905065b997474e3 P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:23.202690 15437 tablet_bootstrap.cc:492] T e0206444ca194f8f98e9d71aa858283d P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:23.202498 14989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0206444ca194f8f98e9d71aa858283d" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:23.203155 15437 ts_tablet_manager.cc:1397] T e0206444ca194f8f98e9d71aa858283d P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.200s user 0.051s sys 0.074s
I20260810 08:00:23.205754 15437 raft_consensus.cc:348] T e0206444ca194f8f98e9d71aa858283d P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:23.206483 15437 raft_consensus.cc:374] T e0206444ca194f8f98e9d71aa858283d P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:23.206786 15437 raft_consensus.cc:729] T e0206444ca194f8f98e9d71aa858283d P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:23.207430 15437 consensus_queue.cc:260] T e0206444ca194f8f98e9d71aa858283d P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:23.208353 14688 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet 3cdca8a5b59a449fb905065b997474e3
I20260810 08:00:23.209566 15437 ts_tablet_manager.cc:1428] T e0206444ca194f8f98e9d71aa858283d P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.006s user 0.000s sys 0.004s
W20260810 08:00:23.214350 15234 leader_election.cc:341] T e0206444ca194f8f98e9d71aa858283d P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:23.216595 14688 catalog_manager.cc:6565] ReplaceTablet: tablet 3cdca8a5b59a449fb905065b997474e3 deleted and replaced by tablet eea24ebbcf3b4f79b8910613f1cad70c
I20260810 08:00:23.225672 15277 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cdca8a5b59a449fb905065b997474e3 with delete_type TABLET_DATA_DELETED (replaced by tablet eea24ebbcf3b4f79b8910613f1cad70c) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:23.228145 15302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cdca8a5b59a449fb905065b997474e3" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
W20260810 08:00:23.233801 14918 leader_election.cc:341] T 3cdca8a5b59a449fb905065b997474e3 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:23.242297 15444 raft_consensus.cc:482] T 3cdca8a5b59a449fb905065b997474e3 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:23.242884 15444 raft_consensus.cc:504] T 3cdca8a5b59a449fb905065b997474e3 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:23.245985 15123 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cdca8a5b59a449fb905065b997474e3 with delete_type TABLET_DATA_DELETED (replaced by tablet eea24ebbcf3b4f79b8910613f1cad70c) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:23.248934 15147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0206444ca194f8f98e9d71aa858283d" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:23.249986 15147 raft_consensus.cc:2454] T e0206444ca194f8f98e9d71aa858283d P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 0.
I20260810 08:00:23.251060 14689 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet b370eb11d5ab4a6e8a253cd0c36d3ca4
I20260810 08:00:23.252157 15444 leader_election.cc:288] T 3cdca8a5b59a449fb905065b997474e3 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:23.252594 14689 catalog_manager.cc:6565] ReplaceTablet: tablet b370eb11d5ab4a6e8a253cd0c36d3ca4 deleted and replaced by tablet 92bbad33a92643e3be3899ace5e29f52
I20260810 08:00:23.253809 14964 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cdca8a5b59a449fb905065b997474e3 with delete_type TABLET_DATA_DELETED (replaced by tablet eea24ebbcf3b4f79b8910613f1cad70c) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:23.255983 15125 tablet_service.cc:1514] Processing DeleteTablet for tablet b370eb11d5ab4a6e8a253cd0c36d3ca4 with delete_type TABLET_DATA_DELETED (replaced by tablet 92bbad33a92643e3be3899ace5e29f52) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:23.254992 15234 leader_election.cc:302] T e0206444ca194f8f98e9d71aa858283d P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, 9d7219686f304f62a54acc7497cbbd07; no voters: a91b27762182454baada71e91ceab224
I20260810 08:00:23.257680 15447 raft_consensus.cc:2781] T e0206444ca194f8f98e9d71aa858283d P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:23.258057 15447 raft_consensus.cc:482] T e0206444ca194f8f98e9d71aa858283d P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:23.258421 15447 raft_consensus.cc:3037] T e0206444ca194f8f98e9d71aa858283d P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:23.263927 15479 tablet_replica.cc:330] T 3cdca8a5b59a449fb905065b997474e3 P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:23.265112 15479 raft_consensus.cc:2229] T 3cdca8a5b59a449fb905065b997474e3 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:23.265102 14966 tablet_service.cc:1514] Processing DeleteTablet for tablet b370eb11d5ab4a6e8a253cd0c36d3ca4 with delete_type TABLET_DATA_DELETED (replaced by tablet 92bbad33a92643e3be3899ace5e29f52) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:23.266978 15482 tablet_replica.cc:330] T 3cdca8a5b59a449fb905065b997474e3 P a91b27762182454baada71e91ceab224: stopping tablet replica
I20260810 08:00:23.267885 15482 raft_consensus.cc:2229] T 3cdca8a5b59a449fb905065b997474e3 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:23.272832 15482 raft_consensus.cc:2258] T 3cdca8a5b59a449fb905065b997474e3 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:23.273676 15479 raft_consensus.cc:2258] T 3cdca8a5b59a449fb905065b997474e3 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:23.276839 15479 ts_tablet_manager.cc:1905] T 3cdca8a5b59a449fb905065b997474e3 P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:23.281860 15482 ts_tablet_manager.cc:1905] T 3cdca8a5b59a449fb905065b997474e3 P a91b27762182454baada71e91ceab224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:23.282855 14989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cdca8a5b59a449fb905065b997474e3" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
W20260810 08:00:23.284631 15079 leader_election.cc:341] T 3cdca8a5b59a449fb905065b997474e3 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: Tablet not RUNNING: STOPPED
I20260810 08:00:23.294600 15147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cdca8a5b59a449fb905065b997474e3" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
W20260810 08:00:23.308099 14922 leader_election.cc:341] T 3cdca8a5b59a449fb905065b997474e3 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Illegal state: Tablet not RUNNING: STOPPED
I20260810 08:00:23.308626 14922 leader_election.cc:302] T 3cdca8a5b59a449fb905065b997474e3 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a91b27762182454baada71e91ceab224; no voters: 3fc5e568f3724f158b616c303efb9eae, 9d7219686f304f62a54acc7497cbbd07
W20260810 08:00:23.309491 14922 raft_consensus.cc:2670] T 3cdca8a5b59a449fb905065b997474e3 P a91b27762182454baada71e91ceab224: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260810 08:00:23.320271 15439 tablet_bootstrap.cc:492] T 3cdca8a5b59a449fb905065b997474e3 P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:23.322193 15439 ts_tablet_manager.cc:1397] T 3cdca8a5b59a449fb905065b997474e3 P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.137s user 0.021s sys 0.004s
I20260810 08:00:23.325189 15439 raft_consensus.cc:348] T 3cdca8a5b59a449fb905065b997474e3 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:23.326104 15439 raft_consensus.cc:374] T 3cdca8a5b59a449fb905065b997474e3 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:23.326381 15439 raft_consensus.cc:729] T 3cdca8a5b59a449fb905065b997474e3 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:23.327104 15439 consensus_queue.cc:260] T 3cdca8a5b59a449fb905065b997474e3 P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:23.317585 15447 raft_consensus.cc:504] T e0206444ca194f8f98e9d71aa858283d P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:23.331117 15479 ts_tablet_manager.cc:1918] T 3cdca8a5b59a449fb905065b997474e3 P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:23.331575 15479 log.cc:1196] T 3cdca8a5b59a449fb905065b997474e3 P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/3cdca8a5b59a449fb905065b997474e3
I20260810 08:00:23.332528 15479 ts_tablet_manager.cc:1939] T 3cdca8a5b59a449fb905065b997474e3 P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:23.333726 15447 leader_election.cc:288] T e0206444ca194f8f98e9d71aa858283d P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Requested vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:23.334692 15479 tablet_replica.cc:330] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:23.335237 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet 3cdca8a5b59a449fb905065b997474e3 (table default.Table_5d982fdf0c924d0ebecb9de16fdebd15 [id=cd26deac07054fd6b8667aae1f1e272b]) successfully deleted
I20260810 08:00:23.335418 15479 raft_consensus.cc:2229] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:23.335886 15479 raft_consensus.cc:2258] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260810 08:00:23.338794 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet 3cdca8a5b59a449fb905065b997474e3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3cdca8a5b59a449fb905065b997474e3 already in progress: creating tablet
I20260810 08:00:23.339542 15479 ts_tablet_manager.cc:1905] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:23.341086 15147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0206444ca194f8f98e9d71aa858283d" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07"
I20260810 08:00:23.341115 14989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0206444ca194f8f98e9d71aa858283d" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224"
I20260810 08:00:23.341697 15147 raft_consensus.cc:3037] T e0206444ca194f8f98e9d71aa858283d P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:23.341832 14989 raft_consensus.cc:3037] T e0206444ca194f8f98e9d71aa858283d P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:23.345994 15277 tablet_service.cc:1514] Processing DeleteTablet for tablet b370eb11d5ab4a6e8a253cd0c36d3ca4 with delete_type TABLET_DATA_DELETED (replaced by tablet 92bbad33a92643e3be3899ace5e29f52) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:23.347128 15478 tablet_replica.cc:330] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P 3fc5e568f3724f158b616c303efb9eae: stopping tablet replica
I20260810 08:00:23.348179 15478 raft_consensus.cc:2229] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:23.348838 15478 raft_consensus.cc:2258] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:23.349555 14989 raft_consensus.cc:2454] T e0206444ca194f8f98e9d71aa858283d P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 1.
I20260810 08:00:23.350757 15234 leader_election.cc:302] T e0206444ca194f8f98e9d71aa858283d P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224; no voters:
I20260810 08:00:23.350920 15302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cdca8a5b59a449fb905065b997474e3" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:23.351579 15302 raft_consensus.cc:2454] T 3cdca8a5b59a449fb905065b997474e3 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d7219686f304f62a54acc7497cbbd07 in term 0.
I20260810 08:00:23.352586 15447 raft_consensus.cc:2781] T e0206444ca194f8f98e9d71aa858283d P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Leader election won for term 1
I20260810 08:00:23.353138 15447 raft_consensus.cc:686] T e0206444ca194f8f98e9d71aa858283d P 3fc5e568f3724f158b616c303efb9eae [term 1 LEADER]: Becoming Leader. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Running, Role: LEADER
I20260810 08:00:23.354215 14691 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_8db9299a79354b7595c1b2522f07cbec" } new_table_name: "default.Table_d3c9526235bd464788ee5b57b1ede9a0" modify_external_catalogs: true
I20260810 08:00:23.355489 15478 ts_tablet_manager.cc:1905] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P 3fc5e568f3724f158b616c303efb9eae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:23.356495 15447 consensus_queue.cc:237] T e0206444ca194f8f98e9d71aa858283d P 3fc5e568f3724f158b616c303efb9eae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:23.363787 14694 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_71132b29527746dd98788028dd7c2016" } new_table_name: "default.Table_dcbe00e111e246a3975b5a4bd8cce1a0" modify_external_catalogs: true
I20260810 08:00:23.366804 15277 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cdca8a5b59a449fb905065b997474e3 with delete_type TABLET_DATA_DELETED (replaced by tablet eea24ebbcf3b4f79b8910613f1cad70c) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:23.368163 15076 leader_election.cc:302] T 3cdca8a5b59a449fb905065b997474e3 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, 9d7219686f304f62a54acc7497cbbd07; no voters: a91b27762182454baada71e91ceab224
W20260810 08:00:23.369347 15076 raft_consensus.cc:2670] T 3cdca8a5b59a449fb905065b997474e3 P 9d7219686f304f62a54acc7497cbbd07: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260810 08:00:23.376320 15147 raft_consensus.cc:2454] T e0206444ca194f8f98e9d71aa858283d P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 1.
I20260810 08:00:23.378621 15479 ts_tablet_manager.cc:1918] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:23.379073 15479 log.cc:1196] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/b370eb11d5ab4a6e8a253cd0c36d3ca4
I20260810 08:00:23.379833 15479 ts_tablet_manager.cc:1939] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:23.384548 15439 ts_tablet_manager.cc:1428] T 3cdca8a5b59a449fb905065b997474e3 P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.062s user 0.006s sys 0.005s
I20260810 08:00:23.395143 14692 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_58c3137b04464b6d845b2a9a753886eb" } modify_external_catalogs: true
I20260810 08:00:23.395944 14692 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_58c3137b04464b6d845b2a9a753886eb" } modify_external_catalogs: true
I20260810 08:00:23.397455 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet b370eb11d5ab4a6e8a253cd0c36d3ca4 (table default.Table_7f3a4e19ded64b3783a00b3d4c6839c3 [id=6c6afcf578534997878b98feb9574e2d]) successfully deleted
I20260810 08:00:23.402426 15482 ts_tablet_manager.cc:1918] T 3cdca8a5b59a449fb905065b997474e3 P a91b27762182454baada71e91ceab224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:23.402895 15482 log.cc:1196] T 3cdca8a5b59a449fb905065b997474e3 P a91b27762182454baada71e91ceab224: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal/wals/3cdca8a5b59a449fb905065b997474e3
I20260810 08:00:23.414379 15482 ts_tablet_manager.cc:1939] T 3cdca8a5b59a449fb905065b997474e3 P a91b27762182454baada71e91ceab224: Deleting consensus metadata
I20260810 08:00:23.418928 15482 tablet_replica.cc:330] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P a91b27762182454baada71e91ceab224: stopping tablet replica
I20260810 08:00:23.419724 15482 raft_consensus.cc:2229] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:23.420215 15482 raft_consensus.cc:2258] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:23.424242 15482 ts_tablet_manager.cc:1905] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P a91b27762182454baada71e91ceab224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:23.425065 14681 catalog_manager.cc:4841] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): tablet 3cdca8a5b59a449fb905065b997474e3 (table default.Table_5d982fdf0c924d0ebecb9de16fdebd15 [id=cd26deac07054fd6b8667aae1f1e272b]) successfully deleted
I20260810 08:00:23.430763 14688 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_0f90d62d68f14a6895531b34d93d7d87" } modify_external_catalogs: true
I20260810 08:00:23.431674 14688 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_0f90d62d68f14a6895531b34d93d7d87" } modify_external_catalogs: true
I20260810 08:00:23.438585 15478 ts_tablet_manager.cc:1918] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P 3fc5e568f3724f158b616c303efb9eae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:23.439189 15478 log.cc:1196] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P 3fc5e568f3724f158b616c303efb9eae: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal/wals/b370eb11d5ab4a6e8a253cd0c36d3ca4
I20260810 08:00:23.440246 15478 ts_tablet_manager.cc:1939] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P 3fc5e568f3724f158b616c303efb9eae: Deleting consensus metadata
I20260810 08:00:23.443495 14678 catalog_manager.cc:4841] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): tablet b370eb11d5ab4a6e8a253cd0c36d3ca4 (table default.Table_7f3a4e19ded64b3783a00b3d4c6839c3 [id=6c6afcf578534997878b98feb9574e2d]) successfully deleted
I20260810 08:00:23.444480 15478 tablet_replica.cc:330] T 3cdca8a5b59a449fb905065b997474e3 P 3fc5e568f3724f158b616c303efb9eae: stopping tablet replica
I20260810 08:00:23.444530 14689 master_service.cc:946] ReplaceTablet: received request to replace tablet b370eb11d5ab4a6e8a253cd0c36d3ca4 from {username='slave'} at 127.0.0.1:42300
I20260810 08:00:23.445586 15478 raft_consensus.cc:2229] T 3cdca8a5b59a449fb905065b997474e3 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:23.446106 15478 raft_consensus.cc:2258] T 3cdca8a5b59a449fb905065b997474e3 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:23.448764 15478 ts_tablet_manager.cc:1905] T 3cdca8a5b59a449fb905065b997474e3 P 3fc5e568f3724f158b616c303efb9eae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:23.449010 14693 catalog_manager.cc:5495] T e0206444ca194f8f98e9d71aa858283d P 3fc5e568f3724f158b616c303efb9eae reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fc5e568f3724f158b616c303efb9eae (127.14.57.195). New cstate: current_term: 1 leader_uuid: "3fc5e568f3724f158b616c303efb9eae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } health_report { overall_health: HEALTHY } } }
I20260810 08:00:23.455341 15277 tablet_service.cc:1467] Processing CreateTablet for tablet eea24ebbcf3b4f79b8910613f1cad70c (DEFAULT_TABLE table=default.Table_5d982fdf0c924d0ebecb9de16fdebd15 [id=cd26deac07054fd6b8667aae1f1e272b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:23.456169 14964 tablet_service.cc:1467] Processing CreateTablet for tablet eea24ebbcf3b4f79b8910613f1cad70c (DEFAULT_TABLE table=default.Table_5d982fdf0c924d0ebecb9de16fdebd15 [id=cd26deac07054fd6b8667aae1f1e272b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:23.456530 15277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eea24ebbcf3b4f79b8910613f1cad70c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:23.457326 14964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eea24ebbcf3b4f79b8910613f1cad70c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:23.457142 15123 tablet_service.cc:1467] Processing CreateTablet for tablet eea24ebbcf3b4f79b8910613f1cad70c (DEFAULT_TABLE table=default.Table_5d982fdf0c924d0ebecb9de16fdebd15 [id=cd26deac07054fd6b8667aae1f1e272b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:23.458288 15123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eea24ebbcf3b4f79b8910613f1cad70c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:23.463500 15280 tablet_service.cc:1467] Processing CreateTablet for tablet 92bbad33a92643e3be3899ace5e29f52 (DEFAULT_TABLE table=default.Table_7f3a4e19ded64b3783a00b3d4c6839c3 [id=6c6afcf578534997878b98feb9574e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:23.464636 15280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92bbad33a92643e3be3899ace5e29f52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:23.467797 14695 master_service.cc:946] ReplaceTablet: received request to replace tablet 06e3130f1b2744d1907390c4f2ce8ce1 from {username='slave'} at 127.0.0.1:42300
I20260810 08:00:23.469022 14966 tablet_service.cc:1467] Processing CreateTablet for tablet 92bbad33a92643e3be3899ace5e29f52 (DEFAULT_TABLE table=default.Table_7f3a4e19ded64b3783a00b3d4c6839c3 [id=6c6afcf578534997878b98feb9574e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:23.470167 14966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92bbad33a92643e3be3899ace5e29f52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:23.483247 15437 tablet_bootstrap.cc:492] T eea24ebbcf3b4f79b8910613f1cad70c P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:23.485271 15482 ts_tablet_manager.cc:1918] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P a91b27762182454baada71e91ceab224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:23.485769 15482 log.cc:1196] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P a91b27762182454baada71e91ceab224: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal/wals/b370eb11d5ab4a6e8a253cd0c36d3ca4
I20260810 08:00:23.488063 15482 ts_tablet_manager.cc:1939] T b370eb11d5ab4a6e8a253cd0c36d3ca4 P a91b27762182454baada71e91ceab224: Deleting consensus metadata
W20260810 08:00:23.488107 14691 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ce99d91e79cf4e06b71110eada58a8e9 has no consensus state
I20260810 08:00:23.491058 14681 catalog_manager.cc:4841] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): tablet b370eb11d5ab4a6e8a253cd0c36d3ca4 (table default.Table_7f3a4e19ded64b3783a00b3d4c6839c3 [id=6c6afcf578534997878b98feb9574e2d]) successfully deleted
I20260810 08:00:23.497093 15437 tablet_bootstrap.cc:654] T eea24ebbcf3b4f79b8910613f1cad70c P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:23.498289 15125 tablet_service.cc:1467] Processing CreateTablet for tablet 92bbad33a92643e3be3899ace5e29f52 (DEFAULT_TABLE table=default.Table_7f3a4e19ded64b3783a00b3d4c6839c3 [id=6c6afcf578534997878b98feb9574e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:23.499374 15125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92bbad33a92643e3be3899ace5e29f52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:23.503255 15478 ts_tablet_manager.cc:1918] T 3cdca8a5b59a449fb905065b997474e3 P 3fc5e568f3724f158b616c303efb9eae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:23.503664 15478 log.cc:1196] T 3cdca8a5b59a449fb905065b997474e3 P 3fc5e568f3724f158b616c303efb9eae: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal/wals/3cdca8a5b59a449fb905065b997474e3
I20260810 08:00:23.504336 15478 ts_tablet_manager.cc:1939] T 3cdca8a5b59a449fb905065b997474e3 P 3fc5e568f3724f158b616c303efb9eae: Deleting consensus metadata
I20260810 08:00:23.506925 14678 catalog_manager.cc:4841] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): tablet 3cdca8a5b59a449fb905065b997474e3 (table default.Table_5d982fdf0c924d0ebecb9de16fdebd15 [id=cd26deac07054fd6b8667aae1f1e272b]) successfully deleted
I20260810 08:00:23.508037 15439 tablet_bootstrap.cc:492] T eea24ebbcf3b4f79b8910613f1cad70c P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:23.515872 15439 tablet_bootstrap.cc:654] T eea24ebbcf3b4f79b8910613f1cad70c P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
W20260810 08:00:23.522219 14694 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a074a741235e4c4b8db1fdb45fb971a8 has no consensus state
I20260810 08:00:23.524199 15439 tablet_bootstrap.cc:492] T eea24ebbcf3b4f79b8910613f1cad70c P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:23.524713 15439 ts_tablet_manager.cc:1397] T eea24ebbcf3b4f79b8910613f1cad70c P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.001s
I20260810 08:00:23.527431 15439 raft_consensus.cc:348] T eea24ebbcf3b4f79b8910613f1cad70c P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:23.528429 15439 raft_consensus.cc:374] T eea24ebbcf3b4f79b8910613f1cad70c P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:23.528878 15439 raft_consensus.cc:729] T eea24ebbcf3b4f79b8910613f1cad70c P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:23.529953 15439 consensus_queue.cc:260] T eea24ebbcf3b4f79b8910613f1cad70c P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:23.532162 14694 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_4008644ee3004c9382c61670dd08b89e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:23.533991 14694 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4008644ee3004c9382c61670dd08b89e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:23.537333 14691 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_e4a35c3d77ba4acfab38e0df4bb74219"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:23.540972 14691 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e4a35c3d77ba4acfab38e0df4bb74219 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:23.542945 15490 tablet_bootstrap.cc:492] T 92bbad33a92643e3be3899ace5e29f52 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:23.545027 15439 ts_tablet_manager.cc:1428] T eea24ebbcf3b4f79b8910613f1cad70c P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
W20260810 08:00:23.545682 14680 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet ce99d91e79cf4e06b71110eada58a8e9 (table default.Table_d3c9526235bd464788ee5b57b1ede9a0 [id=d1c97b8884c844f8b808baaaf7038af8], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet ce99d91e79cf4e06b71110eada58a8e9 has no consensus state
I20260810 08:00:23.548233 15437 tablet_bootstrap.cc:492] T eea24ebbcf3b4f79b8910613f1cad70c P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:23.548810 15437 ts_tablet_manager.cc:1397] T eea24ebbcf3b4f79b8910613f1cad70c P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.066s user 0.010s sys 0.002s
I20260810 08:00:23.552142 15437 raft_consensus.cc:348] T eea24ebbcf3b4f79b8910613f1cad70c P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:23.553148 15437 raft_consensus.cc:374] T eea24ebbcf3b4f79b8910613f1cad70c P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:23.553514 15437 raft_consensus.cc:729] T eea24ebbcf3b4f79b8910613f1cad70c P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:23.554313 15437 consensus_queue.cc:260] T eea24ebbcf3b4f79b8910613f1cad70c P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:23.557102 15437 ts_tablet_manager.cc:1428] T eea24ebbcf3b4f79b8910613f1cad70c P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20260810 08:00:23.558058 15437 tablet_bootstrap.cc:492] T 92bbad33a92643e3be3899ace5e29f52 P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:23.560642 14692 catalog_manager.cc:5777] Not sending DeleteTablet requests; no consensus state for tablet 7914c40923544d0ab3a6db6403599cab
I20260810 08:00:23.564155 15437 tablet_bootstrap.cc:654] T 92bbad33a92643e3be3899ace5e29f52 P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:23.575887 15437 tablet_bootstrap.cc:492] T 92bbad33a92643e3be3899ace5e29f52 P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:23.576388 15437 ts_tablet_manager.cc:1397] T 92bbad33a92643e3be3899ace5e29f52 P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.007s
I20260810 08:00:23.578992 15437 raft_consensus.cc:348] T 92bbad33a92643e3be3899ace5e29f52 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:23.579787 15437 raft_consensus.cc:374] T 92bbad33a92643e3be3899ace5e29f52 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:23.580070 15437 raft_consensus.cc:729] T 92bbad33a92643e3be3899ace5e29f52 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:23.580740 15437 consensus_queue.cc:260] T 92bbad33a92643e3be3899ace5e29f52 P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:23.583339 15437 ts_tablet_manager.cc:1428] T 92bbad33a92643e3be3899ace5e29f52 P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260810 08:00:23.585702 14690 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_dcb3692bb6ec4ad1a6e6c9cffd530482"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:23.587440 14690 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dcb3692bb6ec4ad1a6e6c9cffd530482 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:23.596125 15439 tablet_bootstrap.cc:492] T 92bbad33a92643e3be3899ace5e29f52 P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:23.596443 14688 catalog_manager.cc:5777] Not sending DeleteTablet requests; no consensus state for tablet 198f3abf18e14686af953157370389eb
I20260810 08:00:23.607254 15490 tablet_bootstrap.cc:654] T 92bbad33a92643e3be3899ace5e29f52 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:23.610447 15439 tablet_bootstrap.cc:654] T 92bbad33a92643e3be3899ace5e29f52 P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:23.625896 15439 tablet_bootstrap.cc:492] T 92bbad33a92643e3be3899ace5e29f52 P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:23.626367 15439 ts_tablet_manager.cc:1397] T 92bbad33a92643e3be3899ace5e29f52 P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.009s
I20260810 08:00:23.628937 15439 raft_consensus.cc:348] T 92bbad33a92643e3be3899ace5e29f52 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:23.629693 15439 raft_consensus.cc:374] T 92bbad33a92643e3be3899ace5e29f52 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:23.629976 15439 raft_consensus.cc:729] T 92bbad33a92643e3be3899ace5e29f52 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:23.629760 14688 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_3a39902b550c4390a56429d0ef94860e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:23.631541 14688 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a39902b550c4390a56429d0ef94860e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:23.635644 15439 consensus_queue.cc:260] T 92bbad33a92643e3be3899ace5e29f52 P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:23.648811 15439 ts_tablet_manager.cc:1428] T 92bbad33a92643e3be3899ace5e29f52 P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.022s user 0.007s sys 0.000s
I20260810 08:00:23.648970 15490 tablet_bootstrap.cc:492] T 92bbad33a92643e3be3899ace5e29f52 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:23.649389 15490 ts_tablet_manager.cc:1397] T 92bbad33a92643e3be3899ace5e29f52 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.107s user 0.010s sys 0.011s
I20260810 08:00:23.652163 15490 raft_consensus.cc:348] T 92bbad33a92643e3be3899ace5e29f52 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:23.652907 15490 raft_consensus.cc:374] T 92bbad33a92643e3be3899ace5e29f52 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:23.653190 15490 raft_consensus.cc:729] T 92bbad33a92643e3be3899ace5e29f52 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:23.653980 15490 consensus_queue.cc:260] T 92bbad33a92643e3be3899ace5e29f52 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:23.657313 15490 ts_tablet_manager.cc:1428] T 92bbad33a92643e3be3899ace5e29f52 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260810 08:00:23.667058 14695 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet 06e3130f1b2744d1907390c4f2ce8ce1
I20260810 08:00:23.668633 14695 catalog_manager.cc:6565] ReplaceTablet: tablet 06e3130f1b2744d1907390c4f2ce8ce1 deleted and replaced by tablet 0c152aab7e244afd857534869b13815c
I20260810 08:00:23.675562 15277 tablet_service.cc:1514] Processing DeleteTablet for tablet 06e3130f1b2744d1907390c4f2ce8ce1 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c152aab7e244afd857534869b13815c) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:23.676450 15478 tablet_replica.cc:330] T 06e3130f1b2744d1907390c4f2ce8ce1 P 3fc5e568f3724f158b616c303efb9eae: stopping tablet replica
I20260810 08:00:23.677234 15478 raft_consensus.cc:2229] T 06e3130f1b2744d1907390c4f2ce8ce1 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:23.677728 15478 raft_consensus.cc:2258] T 06e3130f1b2744d1907390c4f2ce8ce1 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:23.680481 15478 ts_tablet_manager.cc:1905] T 06e3130f1b2744d1907390c4f2ce8ce1 P 3fc5e568f3724f158b616c303efb9eae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:23.682902 15125 tablet_service.cc:1514] Processing DeleteTablet for tablet 06e3130f1b2744d1907390c4f2ce8ce1 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c152aab7e244afd857534869b13815c) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:23.684593 14694 master_service.cc:946] ReplaceTablet: received request to replace tablet eea24ebbcf3b4f79b8910613f1cad70c from {username='slave'} at 127.0.0.1:42300
I20260810 08:00:23.684650 14966 tablet_service.cc:1514] Processing DeleteTablet for tablet 06e3130f1b2744d1907390c4f2ce8ce1 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c152aab7e244afd857534869b13815c) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:23.685560 15482 tablet_replica.cc:330] T 06e3130f1b2744d1907390c4f2ce8ce1 P a91b27762182454baada71e91ceab224: stopping tablet replica
I20260810 08:00:23.686362 15482 raft_consensus.cc:2229] T 06e3130f1b2744d1907390c4f2ce8ce1 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:23.686882 15482 raft_consensus.cc:2258] T 06e3130f1b2744d1907390c4f2ce8ce1 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:23.688871 15482 ts_tablet_manager.cc:1905] T 06e3130f1b2744d1907390c4f2ce8ce1 P a91b27762182454baada71e91ceab224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:23.689299 15479 tablet_replica.cc:330] T 06e3130f1b2744d1907390c4f2ce8ce1 P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:23.690322 15479 raft_consensus.cc:2229] T 06e3130f1b2744d1907390c4f2ce8ce1 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:23.690871 15479 raft_consensus.cc:2258] T 06e3130f1b2744d1907390c4f2ce8ce1 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:23.692726 15479 ts_tablet_manager.cc:1905] T 06e3130f1b2744d1907390c4f2ce8ce1 P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:23.699913 14690 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_d3c9526235bd464788ee5b57b1ede9a0" } new_table_name: "default.Table_82804e0d8405498b86dd27530bf9384f" modify_external_catalogs: true
I20260810 08:00:23.711865 15490 tablet_bootstrap.cc:492] T eea24ebbcf3b4f79b8910613f1cad70c P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:23.719813 15482 ts_tablet_manager.cc:1918] T 06e3130f1b2744d1907390c4f2ce8ce1 P a91b27762182454baada71e91ceab224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:23.720319 15482 log.cc:1196] T 06e3130f1b2744d1907390c4f2ce8ce1 P a91b27762182454baada71e91ceab224: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal/wals/06e3130f1b2744d1907390c4f2ce8ce1
I20260810 08:00:23.720746 15490 tablet_bootstrap.cc:654] T eea24ebbcf3b4f79b8910613f1cad70c P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:23.721110 15482 ts_tablet_manager.cc:1939] T 06e3130f1b2744d1907390c4f2ce8ce1 P a91b27762182454baada71e91ceab224: Deleting consensus metadata
I20260810 08:00:23.724829 14681 catalog_manager.cc:4841] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): tablet 06e3130f1b2744d1907390c4f2ce8ce1 (table default.Table_8ac532f4969e45a197db311de2a7b795 [id=7893824f2c374508b7b3c4635ebea5b6]) successfully deleted
I20260810 08:00:23.727674 14688 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_e4a35c3d77ba4acfab38e0df4bb74219" } new_table_name: "default.Table_dc4e4fdad19f4171b118685df23a7d29" modify_external_catalogs: true
I20260810 08:00:23.746063 15478 ts_tablet_manager.cc:1918] T 06e3130f1b2744d1907390c4f2ce8ce1 P 3fc5e568f3724f158b616c303efb9eae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:23.746766 15478 log.cc:1196] T 06e3130f1b2744d1907390c4f2ce8ce1 P 3fc5e568f3724f158b616c303efb9eae: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal/wals/06e3130f1b2744d1907390c4f2ce8ce1
I20260810 08:00:23.748217 15478 ts_tablet_manager.cc:1939] T 06e3130f1b2744d1907390c4f2ce8ce1 P 3fc5e568f3724f158b616c303efb9eae: Deleting consensus metadata
W20260810 08:00:23.750448 14690 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ce99d91e79cf4e06b71110eada58a8e9 has no consensus state
I20260810 08:00:23.751993 14678 catalog_manager.cc:4841] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): tablet 06e3130f1b2744d1907390c4f2ce8ce1 (table default.Table_8ac532f4969e45a197db311de2a7b795 [id=7893824f2c374508b7b3c4635ebea5b6]) successfully deleted
I20260810 08:00:23.757299 15277 tablet_service.cc:1467] Processing CreateTablet for tablet a074a741235e4c4b8db1fdb45fb971a8 (DEFAULT_TABLE table=default.Table_dcbe00e111e246a3975b5a4bd8cce1a0 [id=72cc95ab8de44814aaeece045b4d8d77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:23.758803 15277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a074a741235e4c4b8db1fdb45fb971a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:23.763892 14966 tablet_service.cc:1467] Processing CreateTablet for tablet a074a741235e4c4b8db1fdb45fb971a8 (DEFAULT_TABLE table=default.Table_dcbe00e111e246a3975b5a4bd8cce1a0 [id=72cc95ab8de44814aaeece045b4d8d77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:23.764722 15125 tablet_service.cc:1467] Processing CreateTablet for tablet a074a741235e4c4b8db1fdb45fb971a8 (DEFAULT_TABLE table=default.Table_dcbe00e111e246a3975b5a4bd8cce1a0 [id=72cc95ab8de44814aaeece045b4d8d77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:23.765055 14966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a074a741235e4c4b8db1fdb45fb971a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:23.765203 15123 tablet_service.cc:1467] Processing CreateTablet for tablet ce99d91e79cf4e06b71110eada58a8e9 (DEFAULT_TABLE table=default.Table_82804e0d8405498b86dd27530bf9384f [id=d1c97b8884c844f8b808baaaf7038af8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:23.766460 14964 tablet_service.cc:1467] Processing CreateTablet for tablet ce99d91e79cf4e06b71110eada58a8e9 (DEFAULT_TABLE table=default.Table_82804e0d8405498b86dd27530bf9384f [id=d1c97b8884c844f8b808baaaf7038af8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:23.767925 14964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce99d91e79cf4e06b71110eada58a8e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:23.770951 15123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce99d91e79cf4e06b71110eada58a8e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:23.771039 15479 ts_tablet_manager.cc:1918] T 06e3130f1b2744d1907390c4f2ce8ce1 P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:23.771492 15479 log.cc:1196] T 06e3130f1b2744d1907390c4f2ce8ce1 P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/06e3130f1b2744d1907390c4f2ce8ce1
I20260810 08:00:23.771898 14690 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_dcbe00e111e246a3975b5a4bd8cce1a0" } modify_external_catalogs: true
I20260810 08:00:23.772197 15479 ts_tablet_manager.cc:1939] T 06e3130f1b2744d1907390c4f2ce8ce1 P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:23.772234 15280 tablet_service.cc:1467] Processing CreateTablet for tablet ce99d91e79cf4e06b71110eada58a8e9 (DEFAULT_TABLE table=default.Table_82804e0d8405498b86dd27530bf9384f [id=d1c97b8884c844f8b808baaaf7038af8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:23.772497 14690 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_dcbe00e111e246a3975b5a4bd8cce1a0" } modify_external_catalogs: true
I20260810 08:00:23.773789 15280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce99d91e79cf4e06b71110eada58a8e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:23.774257 15125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a074a741235e4c4b8db1fdb45fb971a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:23.775820 15490 tablet_bootstrap.cc:492] T eea24ebbcf3b4f79b8910613f1cad70c P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:23.776244 15490 ts_tablet_manager.cc:1397] T eea24ebbcf3b4f79b8910613f1cad70c P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.065s user 0.008s sys 0.018s
I20260810 08:00:23.778769 15490 raft_consensus.cc:348] T eea24ebbcf3b4f79b8910613f1cad70c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:23.779498 15490 raft_consensus.cc:374] T eea24ebbcf3b4f79b8910613f1cad70c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:23.779770 15490 raft_consensus.cc:729] T eea24ebbcf3b4f79b8910613f1cad70c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:23.780478 15490 consensus_queue.cc:260] T eea24ebbcf3b4f79b8910613f1cad70c P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:23.785769 15490 ts_tablet_manager.cc:1428] T eea24ebbcf3b4f79b8910613f1cad70c P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.009s user 0.007s sys 0.001s
W20260810 08:00:23.794416 14688 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5e0e4f1c76f843ea9000ed25acec73d3 has no consensus state
I20260810 08:00:23.799518 14688 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_82804e0d8405498b86dd27530bf9384f" } modify_external_catalogs: true
I20260810 08:00:23.800122 14688 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_82804e0d8405498b86dd27530bf9384f" } modify_external_catalogs: true
I20260810 08:00:23.802364 14694 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet eea24ebbcf3b4f79b8910613f1cad70c
I20260810 08:00:23.805927 15279 tablet_service.cc:1514] Processing DeleteTablet for tablet eea24ebbcf3b4f79b8910613f1cad70c with delete_type TABLET_DATA_DELETED (replaced by tablet b798daf216454a4598437947928803d1) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:23.806713 15478 tablet_replica.cc:330] T eea24ebbcf3b4f79b8910613f1cad70c P 3fc5e568f3724f158b616c303efb9eae: stopping tablet replica
I20260810 08:00:23.807539 15478 raft_consensus.cc:2229] T eea24ebbcf3b4f79b8910613f1cad70c P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:23.808007 15478 raft_consensus.cc:2258] T eea24ebbcf3b4f79b8910613f1cad70c P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:23.810277 15478 ts_tablet_manager.cc:1905] T eea24ebbcf3b4f79b8910613f1cad70c P 3fc5e568f3724f158b616c303efb9eae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:23.811390 14694 catalog_manager.cc:6565] ReplaceTablet: tablet eea24ebbcf3b4f79b8910613f1cad70c deleted and replaced by tablet b798daf216454a4598437947928803d1
I20260810 08:00:23.813016 14965 tablet_service.cc:1514] Processing DeleteTablet for tablet eea24ebbcf3b4f79b8910613f1cad70c with delete_type TABLET_DATA_DELETED (replaced by tablet b798daf216454a4598437947928803d1) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:23.813779 15122 tablet_service.cc:1514] Processing DeleteTablet for tablet eea24ebbcf3b4f79b8910613f1cad70c with delete_type TABLET_DATA_DELETED (replaced by tablet b798daf216454a4598437947928803d1) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:23.814554 15482 tablet_replica.cc:330] T eea24ebbcf3b4f79b8910613f1cad70c P a91b27762182454baada71e91ceab224: stopping tablet replica
I20260810 08:00:23.815385 15482 raft_consensus.cc:2229] T eea24ebbcf3b4f79b8910613f1cad70c P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:23.815970 15482 raft_consensus.cc:2258] T eea24ebbcf3b4f79b8910613f1cad70c P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:23.818037 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet 06e3130f1b2744d1907390c4f2ce8ce1 (table default.Table_8ac532f4969e45a197db311de2a7b795 [id=7893824f2c374508b7b3c4635ebea5b6]) successfully deleted
I20260810 08:00:23.818403 15482 ts_tablet_manager.cc:1905] T eea24ebbcf3b4f79b8910613f1cad70c P a91b27762182454baada71e91ceab224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:23.819052 15479 tablet_replica.cc:330] T eea24ebbcf3b4f79b8910613f1cad70c P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:23.819880 15479 raft_consensus.cc:2229] T eea24ebbcf3b4f79b8910613f1cad70c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:23.820405 15479 raft_consensus.cc:2258] T eea24ebbcf3b4f79b8910613f1cad70c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:23.827101 15490 tablet_bootstrap.cc:492] T ce99d91e79cf4e06b71110eada58a8e9 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:23.837226 15479 ts_tablet_manager.cc:1905] T eea24ebbcf3b4f79b8910613f1cad70c P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:23.833566 15490 tablet_bootstrap.cc:654] T ce99d91e79cf4e06b71110eada58a8e9 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:23.854773 15482 ts_tablet_manager.cc:1918] T eea24ebbcf3b4f79b8910613f1cad70c P a91b27762182454baada71e91ceab224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:23.855188 15482 log.cc:1196] T eea24ebbcf3b4f79b8910613f1cad70c P a91b27762182454baada71e91ceab224: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal/wals/eea24ebbcf3b4f79b8910613f1cad70c
I20260810 08:00:23.855253 15123 tablet_service.cc:1467] Processing CreateTablet for tablet abdd2b119a4b448dacdd2d87da34832f (DEFAULT_TABLE table=default.Table_3a39902b550c4390a56429d0ef94860e [id=cee29bc23d044a4bae553990b0d3c32c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:23.855863 15482 ts_tablet_manager.cc:1939] T eea24ebbcf3b4f79b8910613f1cad70c P a91b27762182454baada71e91ceab224: Deleting consensus metadata
I20260810 08:00:23.856467 15123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdd2b119a4b448dacdd2d87da34832f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:23.858563 14681 catalog_manager.cc:4841] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): tablet eea24ebbcf3b4f79b8910613f1cad70c (table default.Table_5d982fdf0c924d0ebecb9de16fdebd15 [id=cd26deac07054fd6b8667aae1f1e272b]) successfully deleted
I20260810 08:00:23.859567 14965 tablet_service.cc:1467] Processing CreateTablet for tablet abdd2b119a4b448dacdd2d87da34832f (DEFAULT_TABLE table=default.Table_3a39902b550c4390a56429d0ef94860e [id=cee29bc23d044a4bae553990b0d3c32c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:23.860600 14965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdd2b119a4b448dacdd2d87da34832f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:23.862856 15279 tablet_service.cc:1467] Processing CreateTablet for tablet abdd2b119a4b448dacdd2d87da34832f (DEFAULT_TABLE table=default.Table_3a39902b550c4390a56429d0ef94860e [id=cee29bc23d044a4bae553990b0d3c32c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:23.864001 15279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdd2b119a4b448dacdd2d87da34832f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:23.869908 14967 tablet_service.cc:1467] Processing CreateTablet for tablet 5e0e4f1c76f843ea9000ed25acec73d3 (DEFAULT_TABLE table=default.Table_dc4e4fdad19f4171b118685df23a7d29 [id=9cbd826186904ef687327cd9e02fdc1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:23.871001 14967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e0e4f1c76f843ea9000ed25acec73d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:23.872035 15278 tablet_service.cc:1467] Processing CreateTablet for tablet 5e0e4f1c76f843ea9000ed25acec73d3 (DEFAULT_TABLE table=default.Table_dc4e4fdad19f4171b118685df23a7d29 [id=9cbd826186904ef687327cd9e02fdc1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:23.873128 15278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e0e4f1c76f843ea9000ed25acec73d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:23.874629 15122 tablet_service.cc:1467] Processing CreateTablet for tablet dc3ea02864d040c5bcde38b1a5609676 (DEFAULT_TABLE table=default.Table_4008644ee3004c9382c61670dd08b89e [id=becf1f7969f041d2a6c986f8918187a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:23.875741 15122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc3ea02864d040c5bcde38b1a5609676. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:23.888811 15125 tablet_service.cc:1467] Processing CreateTablet for tablet 5e0e4f1c76f843ea9000ed25acec73d3 (DEFAULT_TABLE table=default.Table_dc4e4fdad19f4171b118685df23a7d29 [id=9cbd826186904ef687327cd9e02fdc1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:23.890112 15125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e0e4f1c76f843ea9000ed25acec73d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:23.890781 14694 master_service.cc:946] ReplaceTablet: received request to replace tablet e0206444ca194f8f98e9d71aa858283d from {username='slave'} at 127.0.0.1:42300
I20260810 08:00:23.894207 14691 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_491d9caa5f6d4e3cb934a5d0582b519c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:23.895114 14693 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_d7227a2f586b4fd29babdc0ae25a0328"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:23.896798 14693 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d7227a2f586b4fd29babdc0ae25a0328 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 08:00:23.900229 14691 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_491d9caa5f6d4e3cb934a5d0582b519c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:23.901386 14692 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_3630f6ea49e84b48a377bad128b8c436"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:23.903115 14692 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3630f6ea49e84b48a377bad128b8c436 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:23.909776 14690 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet a074a741235e4c4b8db1fdb45fb971a8
I20260810 08:00:23.912442 15276 tablet_service.cc:1514] Processing DeleteTablet for tablet a074a741235e4c4b8db1fdb45fb971a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:23 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:23.912914 15123 tablet_service.cc:1514] Processing DeleteTablet for tablet a074a741235e4c4b8db1fdb45fb971a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:23 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:23.913645 14963 tablet_service.cc:1514] Processing DeleteTablet for tablet a074a741235e4c4b8db1fdb45fb971a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:23 UTC) from {username='slave'} at 127.0.0.1:34630
W20260810 08:00:23.917084 14681 catalog_manager.cc:4808] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet a074a741235e4c4b8db1fdb45fb971a8 because the tablet was not found. No further retry: Not found: Tablet not found: a074a741235e4c4b8db1fdb45fb971a8
I20260810 08:00:23.918219 15490 tablet_bootstrap.cc:492] T ce99d91e79cf4e06b71110eada58a8e9 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:23.918666 15490 ts_tablet_manager.cc:1397] T ce99d91e79cf4e06b71110eada58a8e9 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.092s user 0.017s sys 0.016s
I20260810 08:00:23.921406 15123 tablet_service.cc:1467] Processing CreateTablet for tablet aeb7c1c4bb1f45f9b46fe5bd78e96a20 (DEFAULT_TABLE table=default.Table_dcb3692bb6ec4ad1a6e6c9cffd530482 [id=40cf37d4ad57413fa64ee743a5c160c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:23.922585 15123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeb7c1c4bb1f45f9b46fe5bd78e96a20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:23.926435 15276 tablet_service.cc:1467] Processing CreateTablet for tablet dc3ea02864d040c5bcde38b1a5609676 (DEFAULT_TABLE table=default.Table_4008644ee3004c9382c61670dd08b89e [id=becf1f7969f041d2a6c986f8918187a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:23.926728 14963 tablet_service.cc:1467] Processing CreateTablet for tablet dc3ea02864d040c5bcde38b1a5609676 (DEFAULT_TABLE table=default.Table_4008644ee3004c9382c61670dd08b89e [id=becf1f7969f041d2a6c986f8918187a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:23.927599 15276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc3ea02864d040c5bcde38b1a5609676. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:23.927896 14963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc3ea02864d040c5bcde38b1a5609676. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:23.934430 15437 tablet_bootstrap.cc:492] T abdd2b119a4b448dacdd2d87da34832f P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:23.930299 15490 raft_consensus.cc:348] T ce99d91e79cf4e06b71110eada58a8e9 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:23.934926 15490 raft_consensus.cc:374] T ce99d91e79cf4e06b71110eada58a8e9 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:23.935215 15490 raft_consensus.cc:729] T ce99d91e79cf4e06b71110eada58a8e9 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:23.935883 15490 consensus_queue.cc:260] T ce99d91e79cf4e06b71110eada58a8e9 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:23.940467 15437 tablet_bootstrap.cc:654] T abdd2b119a4b448dacdd2d87da34832f P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:23.942692 15490 ts_tablet_manager.cc:1428] T ce99d91e79cf4e06b71110eada58a8e9 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.024s user 0.005s sys 0.003s
I20260810 08:00:23.944059 15490 tablet_bootstrap.cc:492] T a074a741235e4c4b8db1fdb45fb971a8 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:23.956478 15490 tablet_bootstrap.cc:654] T a074a741235e4c4b8db1fdb45fb971a8 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:23.958845 15275 tablet_service.cc:1467] Processing CreateTablet for tablet aeb7c1c4bb1f45f9b46fe5bd78e96a20 (DEFAULT_TABLE table=default.Table_dcb3692bb6ec4ad1a6e6c9cffd530482 [id=40cf37d4ad57413fa64ee743a5c160c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:23.960055 15275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeb7c1c4bb1f45f9b46fe5bd78e96a20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:23.962560 15274 tablet_service.cc:1467] Processing CreateTablet for tablet 0c152aab7e244afd857534869b13815c (DEFAULT_TABLE table=default.Table_8ac532f4969e45a197db311de2a7b795 [id=7893824f2c374508b7b3c4635ebea5b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:23.963626 15274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c152aab7e244afd857534869b13815c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:23.966109 15123 tablet_service.cc:1467] Processing CreateTablet for tablet 0c152aab7e244afd857534869b13815c (DEFAULT_TABLE table=default.Table_8ac532f4969e45a197db311de2a7b795 [id=7893824f2c374508b7b3c4635ebea5b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:23.964449 14965 tablet_service.cc:1467] Processing CreateTablet for tablet aeb7c1c4bb1f45f9b46fe5bd78e96a20 (DEFAULT_TABLE table=default.Table_dcb3692bb6ec4ad1a6e6c9cffd530482 [id=40cf37d4ad57413fa64ee743a5c160c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:23.967255 15123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c152aab7e244afd857534869b13815c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:23.967569 14965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeb7c1c4bb1f45f9b46fe5bd78e96a20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:23.966363 14967 tablet_service.cc:1467] Processing CreateTablet for tablet 0c152aab7e244afd857534869b13815c (DEFAULT_TABLE table=default.Table_8ac532f4969e45a197db311de2a7b795 [id=7893824f2c374508b7b3c4635ebea5b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:23.970209 14967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c152aab7e244afd857534869b13815c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:23.985695 15490 tablet_bootstrap.cc:492] T a074a741235e4c4b8db1fdb45fb971a8 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:23.986375 15490 ts_tablet_manager.cc:1397] T a074a741235e4c4b8db1fdb45fb971a8 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.043s user 0.013s sys 0.004s
I20260810 08:00:23.989337 15490 raft_consensus.cc:348] T a074a741235e4c4b8db1fdb45fb971a8 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:23.990653 15490 raft_consensus.cc:374] T a074a741235e4c4b8db1fdb45fb971a8 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:23.990948 15490 raft_consensus.cc:729] T a074a741235e4c4b8db1fdb45fb971a8 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:23.997504 15478 ts_tablet_manager.cc:1918] T eea24ebbcf3b4f79b8910613f1cad70c P 3fc5e568f3724f158b616c303efb9eae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:23.997922 15478 log.cc:1196] T eea24ebbcf3b4f79b8910613f1cad70c P 3fc5e568f3724f158b616c303efb9eae: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal/wals/eea24ebbcf3b4f79b8910613f1cad70c
I20260810 08:00:23.998588 15478 ts_tablet_manager.cc:1939] T eea24ebbcf3b4f79b8910613f1cad70c P 3fc5e568f3724f158b616c303efb9eae: Deleting consensus metadata
I20260810 08:00:23.999990 15437 tablet_bootstrap.cc:492] T abdd2b119a4b448dacdd2d87da34832f P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:24.000469 15437 ts_tablet_manager.cc:1397] T abdd2b119a4b448dacdd2d87da34832f P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.066s user 0.005s sys 0.005s
I20260810 08:00:24.001127 14678 catalog_manager.cc:4841] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): tablet eea24ebbcf3b4f79b8910613f1cad70c (table default.Table_5d982fdf0c924d0ebecb9de16fdebd15 [id=cd26deac07054fd6b8667aae1f1e272b]) successfully deleted
I20260810 08:00:24.002866 15437 raft_consensus.cc:348] T abdd2b119a4b448dacdd2d87da34832f P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.003505 15437 raft_consensus.cc:374] T abdd2b119a4b448dacdd2d87da34832f P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:24.003731 15437 raft_consensus.cc:729] T abdd2b119a4b448dacdd2d87da34832f P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:24.004280 15437 consensus_queue.cc:260] T abdd2b119a4b448dacdd2d87da34832f P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:23.992590 15490 consensus_queue.cc:260] T a074a741235e4c4b8db1fdb45fb971a8 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.009779 15439 tablet_bootstrap.cc:492] T dc3ea02864d040c5bcde38b1a5609676 P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:24.012199 15490 ts_tablet_manager.cc:1428] T a074a741235e4c4b8db1fdb45fb971a8 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.025s user 0.006s sys 0.000s
I20260810 08:00:24.015852 15439 tablet_bootstrap.cc:654] T dc3ea02864d040c5bcde38b1a5609676 P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
W20260810 08:00:24.020627 15454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 08:00:24.021514 15454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 08:00:24.022302 15454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 08:00:24.023021 14690 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_2e46207a762e4a8f9054bda89fd0293f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:24.023557 14689 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_dc4e4fdad19f4171b118685df23a7d29" } new_table_name: "default.Table_43a063da4167472198ae9533501169eb" modify_external_catalogs: true
I20260810 08:00:24.025426 14686 master_service.cc:946] ReplaceTablet: received request to replace tablet ce99d91e79cf4e06b71110eada58a8e9 from {username='slave'} at 127.0.0.1:42300
W20260810 08:00:24.026087 14690 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e46207a762e4a8f9054bda89fd0293f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:24.027303 14687 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_dcb3692bb6ec4ad1a6e6c9cffd530482" } new_table_name: "default.Table_9b0d38b98f3841828a00a23dda57ae6e" modify_external_catalogs: true
I20260810 08:00:24.039211 15437 ts_tablet_manager.cc:1428] T abdd2b119a4b448dacdd2d87da34832f P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.038s user 0.004s sys 0.000s
I20260810 08:00:24.047276 15437 tablet_bootstrap.cc:492] T 5e0e4f1c76f843ea9000ed25acec73d3 P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:24.049896 15439 tablet_bootstrap.cc:492] T dc3ea02864d040c5bcde38b1a5609676 P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:24.050509 15439 ts_tablet_manager.cc:1397] T dc3ea02864d040c5bcde38b1a5609676 P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.041s user 0.006s sys 0.005s
I20260810 08:00:24.050599 15490 tablet_bootstrap.cc:492] T abdd2b119a4b448dacdd2d87da34832f P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:24.052881 15437 tablet_bootstrap.cc:654] T 5e0e4f1c76f843ea9000ed25acec73d3 P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:24.053195 15439 raft_consensus.cc:348] T dc3ea02864d040c5bcde38b1a5609676 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.054066 15439 raft_consensus.cc:374] T dc3ea02864d040c5bcde38b1a5609676 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:24.054384 15439 raft_consensus.cc:729] T dc3ea02864d040c5bcde38b1a5609676 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:24.055126 15439 consensus_queue.cc:260] T dc3ea02864d040c5bcde38b1a5609676 P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.058686 14694 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet e0206444ca194f8f98e9d71aa858283d
I20260810 08:00:24.061216 15123 tablet_service.cc:1514] Processing DeleteTablet for tablet e0206444ca194f8f98e9d71aa858283d with delete_type TABLET_DATA_DELETED (replaced by tablet 9ddae63b29c34e1397c5765e52e1478f) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:24.061245 14966 tablet_service.cc:1514] Processing DeleteTablet for tablet e0206444ca194f8f98e9d71aa858283d with delete_type TABLET_DATA_DELETED (replaced by tablet 9ddae63b29c34e1397c5765e52e1478f) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:24.062034 15482 tablet_replica.cc:330] T e0206444ca194f8f98e9d71aa858283d P a91b27762182454baada71e91ceab224: stopping tablet replica
I20260810 08:00:24.062780 15482 raft_consensus.cc:2229] T e0206444ca194f8f98e9d71aa858283d P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:24.063215 15482 raft_consensus.cc:2258] T e0206444ca194f8f98e9d71aa858283d P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:24.065009 15482 ts_tablet_manager.cc:1905] T e0206444ca194f8f98e9d71aa858283d P a91b27762182454baada71e91ceab224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:24.065886 14694 catalog_manager.cc:6565] ReplaceTablet: tablet e0206444ca194f8f98e9d71aa858283d deleted and replaced by tablet 9ddae63b29c34e1397c5765e52e1478f
I20260810 08:00:24.072484 15490 tablet_bootstrap.cc:654] T abdd2b119a4b448dacdd2d87da34832f P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:24.092324 15490 tablet_bootstrap.cc:492] T abdd2b119a4b448dacdd2d87da34832f P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:24.092793 15490 ts_tablet_manager.cc:1397] T abdd2b119a4b448dacdd2d87da34832f P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.007s
I20260810 08:00:24.095366 15490 raft_consensus.cc:348] T abdd2b119a4b448dacdd2d87da34832f P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.096105 15490 raft_consensus.cc:374] T abdd2b119a4b448dacdd2d87da34832f P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:24.096381 15490 raft_consensus.cc:729] T abdd2b119a4b448dacdd2d87da34832f P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:24.071882 15275 tablet_service.cc:1514] Processing DeleteTablet for tablet e0206444ca194f8f98e9d71aa858283d with delete_type TABLET_DATA_DELETED (replaced by tablet 9ddae63b29c34e1397c5765e52e1478f) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:24.100828 15490 consensus_queue.cc:260] T abdd2b119a4b448dacdd2d87da34832f P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.110831 15490 ts_tablet_manager.cc:1428] T abdd2b119a4b448dacdd2d87da34832f P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20260810 08:00:24.112743 15439 ts_tablet_manager.cc:1428] T dc3ea02864d040c5bcde38b1a5609676 P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.062s user 0.005s sys 0.000s
I20260810 08:00:24.115814 14694 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_3a39902b550c4390a56429d0ef94860e" } modify_external_catalogs: true
I20260810 08:00:24.116478 14694 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_3a39902b550c4390a56429d0ef94860e" } modify_external_catalogs: true
W20260810 08:00:24.130016 14689 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5e0e4f1c76f843ea9000ed25acec73d3 consensus state has no leader
I20260810 08:00:24.130260 15439 tablet_bootstrap.cc:492] T ce99d91e79cf4e06b71110eada58a8e9 P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:24.134853 14688 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet ce99d91e79cf4e06b71110eada58a8e9
I20260810 08:00:24.137609 15275 tablet_service.cc:1514] Processing DeleteTablet for tablet ce99d91e79cf4e06b71110eada58a8e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:23 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:24.143743 15125 tablet_service.cc:1514] Processing DeleteTablet for tablet ce99d91e79cf4e06b71110eada58a8e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:23 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:24.144165 15123 tablet_service.cc:1467] Processing CreateTablet for tablet b798daf216454a4598437947928803d1 (DEFAULT_TABLE table=default.Table_5d982fdf0c924d0ebecb9de16fdebd15 [id=cd26deac07054fd6b8667aae1f1e272b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:24.144601 15275 tablet_service.cc:1467] Processing CreateTablet for tablet b798daf216454a4598437947928803d1 (DEFAULT_TABLE table=default.Table_5d982fdf0c924d0ebecb9de16fdebd15 [id=cd26deac07054fd6b8667aae1f1e272b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:24.144729 14965 tablet_service.cc:1467] Processing CreateTablet for tablet b798daf216454a4598437947928803d1 (DEFAULT_TABLE table=default.Table_5d982fdf0c924d0ebecb9de16fdebd15 [id=cd26deac07054fd6b8667aae1f1e272b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:24.145323 15123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b798daf216454a4598437947928803d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:24.145723 15275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b798daf216454a4598437947928803d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:24.145823 14965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b798daf216454a4598437947928803d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:24.153872 14967 tablet_service.cc:1514] Processing DeleteTablet for tablet ce99d91e79cf4e06b71110eada58a8e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:23 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:24.155726 15437 tablet_bootstrap.cc:492] T 5e0e4f1c76f843ea9000ed25acec73d3 P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:24.156116 15437 ts_tablet_manager.cc:1397] T 5e0e4f1c76f843ea9000ed25acec73d3 P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.116s user 0.014s sys 0.018s
I20260810 08:00:24.158852 15437 raft_consensus.cc:348] T 5e0e4f1c76f843ea9000ed25acec73d3 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.159607 15437 raft_consensus.cc:374] T 5e0e4f1c76f843ea9000ed25acec73d3 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:24.159885 15437 raft_consensus.cc:729] T 5e0e4f1c76f843ea9000ed25acec73d3 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:24.160524 15437 consensus_queue.cc:260] T 5e0e4f1c76f843ea9000ed25acec73d3 P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.162766 15437 ts_tablet_manager.cc:1428] T 5e0e4f1c76f843ea9000ed25acec73d3 P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 08:00:24.163637 15437 tablet_bootstrap.cc:492] T ce99d91e79cf4e06b71110eada58a8e9 P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:24.170598 15482 ts_tablet_manager.cc:1918] T e0206444ca194f8f98e9d71aa858283d P a91b27762182454baada71e91ceab224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:24.171002 15482 log.cc:1196] T e0206444ca194f8f98e9d71aa858283d P a91b27762182454baada71e91ceab224: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal/wals/e0206444ca194f8f98e9d71aa858283d
I20260810 08:00:24.174266 15437 tablet_bootstrap.cc:654] T ce99d91e79cf4e06b71110eada58a8e9 P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
W20260810 08:00:24.177304 14687 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aeb7c1c4bb1f45f9b46fe5bd78e96a20 consensus state has no leader
W20260810 08:00:24.178869 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet a074a741235e4c4b8db1fdb45fb971a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a074a741235e4c4b8db1fdb45fb971a8 already in progress: creating tablet
I20260810 08:00:24.179840 15478 tablet_replica.cc:330] T e0206444ca194f8f98e9d71aa858283d P 3fc5e568f3724f158b616c303efb9eae: stopping tablet replica
I20260810 08:00:24.180744 15478 raft_consensus.cc:2229] T e0206444ca194f8f98e9d71aa858283d P 3fc5e568f3724f158b616c303efb9eae [term 1 LEADER]: Raft consensus shutting down.
I20260810 08:00:24.181581 15478 pending_rounds.cc:62] T e0206444ca194f8f98e9d71aa858283d P 3fc5e568f3724f158b616c303efb9eae: Trying to abort 1 pending ops.
I20260810 08:00:24.186622 15439 tablet_bootstrap.cc:654] T ce99d91e79cf4e06b71110eada58a8e9 P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:24.161449 15490 tablet_bootstrap.cc:492] T dc3ea02864d040c5bcde38b1a5609676 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:24.190157 15482 ts_tablet_manager.cc:1939] T e0206444ca194f8f98e9d71aa858283d P a91b27762182454baada71e91ceab224: Deleting consensus metadata
I20260810 08:00:24.192858 14681 catalog_manager.cc:4841] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): tablet e0206444ca194f8f98e9d71aa858283d (table default.Table_0e30c621046242609c9f6fc73b16d0e7 [id=e9608d66a9964bc6aafdf84f574fba02]) successfully deleted
W20260810 08:00:24.193902 14681 catalog_manager.cc:4821] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet ce99d91e79cf4e06b71110eada58a8e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce99d91e79cf4e06b71110eada58a8e9 already in progress: creating tablet
I20260810 08:00:24.201670 15490 tablet_bootstrap.cc:654] T dc3ea02864d040c5bcde38b1a5609676 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:24.203401 14694 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet abdd2b119a4b448dacdd2d87da34832f
I20260810 08:00:24.215688 15447 raft_consensus.cc:482] T 92bbad33a92643e3be3899ace5e29f52 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:24.216114 15275 tablet_service.cc:1514] Processing DeleteTablet for tablet abdd2b119a4b448dacdd2d87da34832f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:24 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:24.216296 15447 raft_consensus.cc:504] T 92bbad33a92643e3be3899ace5e29f52 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.181926 15478 pending_rounds.cc:68] T e0206444ca194f8f98e9d71aa858283d P 3fc5e568f3724f158b616c303efb9eae: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316884780521984000 op_type: NO_OP noop_request { }
I20260810 08:00:24.218092 15478 raft_consensus.cc:2866] T e0206444ca194f8f98e9d71aa858283d P 3fc5e568f3724f158b616c303efb9eae [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260810 08:00:24.218581 15478 raft_consensus.cc:2258] T e0206444ca194f8f98e9d71aa858283d P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:24.218840 15447 leader_election.cc:288] T 92bbad33a92643e3be3899ace5e29f52 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:24.222935 14989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bbad33a92643e3be3899ace5e29f52" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:24.223536 14989 raft_consensus.cc:2454] T 92bbad33a92643e3be3899ace5e29f52 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 0.
I20260810 08:00:24.224685 15234 leader_election.cc:302] T 92bbad33a92643e3be3899ace5e29f52 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224; no voters:
I20260810 08:00:24.225832 15447 raft_consensus.cc:2781] T 92bbad33a92643e3be3899ace5e29f52 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:24.226267 15447 raft_consensus.cc:482] T 92bbad33a92643e3be3899ace5e29f52 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:24.226666 15447 raft_consensus.cc:3037] T 92bbad33a92643e3be3899ace5e29f52 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:24.205204 15125 tablet_service.cc:1514] Processing DeleteTablet for tablet abdd2b119a4b448dacdd2d87da34832f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:24 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:24.228601 15478 ts_tablet_manager.cc:1905] T e0206444ca194f8f98e9d71aa858283d P 3fc5e568f3724f158b616c303efb9eae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:24.237305 15437 tablet_bootstrap.cc:492] T ce99d91e79cf4e06b71110eada58a8e9 P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:24.237764 15437 ts_tablet_manager.cc:1397] T ce99d91e79cf4e06b71110eada58a8e9 P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.074s user 0.012s sys 0.015s
I20260810 08:00:24.208503 15479 ts_tablet_manager.cc:1918] T eea24ebbcf3b4f79b8910613f1cad70c P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:24.221583 15147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bbad33a92643e3be3899ace5e29f52" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:24.238389 15147 raft_consensus.cc:2454] T 92bbad33a92643e3be3899ace5e29f52 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 0.
I20260810 08:00:24.239604 15479 log.cc:1196] T eea24ebbcf3b4f79b8910613f1cad70c P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/eea24ebbcf3b4f79b8910613f1cad70c
I20260810 08:00:24.240127 14965 tablet_service.cc:1514] Processing DeleteTablet for tablet abdd2b119a4b448dacdd2d87da34832f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:24 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:24.239991 15437 raft_consensus.cc:348] T ce99d91e79cf4e06b71110eada58a8e9 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.240669 15437 raft_consensus.cc:374] T ce99d91e79cf4e06b71110eada58a8e9 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:24.240829 15482 tablet_replica.cc:330] T abdd2b119a4b448dacdd2d87da34832f P a91b27762182454baada71e91ceab224: stopping tablet replica
I20260810 08:00:24.240952 15437 raft_consensus.cc:729] T ce99d91e79cf4e06b71110eada58a8e9 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:24.241551 15482 raft_consensus.cc:2229] T abdd2b119a4b448dacdd2d87da34832f P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:24.242051 15482 raft_consensus.cc:2258] T abdd2b119a4b448dacdd2d87da34832f P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:24.243584 15439 tablet_bootstrap.cc:492] T ce99d91e79cf4e06b71110eada58a8e9 P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:24.244076 15439 ts_tablet_manager.cc:1397] T ce99d91e79cf4e06b71110eada58a8e9 P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.114s user 0.007s sys 0.003s
I20260810 08:00:24.244194 15482 ts_tablet_manager.cc:1905] T abdd2b119a4b448dacdd2d87da34832f P a91b27762182454baada71e91ceab224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:24.248848 15479 ts_tablet_manager.cc:1939] T eea24ebbcf3b4f79b8910613f1cad70c P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:24.255934 14692 catalog_manager.cc:5587] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601) does not have the latest schema for tablet 5e0e4f1c76f843ea9000ed25acec73d3 (table default.Table_43a063da4167472198ae9533501169eb [id=9cbd826186904ef687327cd9e02fdc1a]). Expected version 2 got 0
I20260810 08:00:24.257794 15447 raft_consensus.cc:504] T 92bbad33a92643e3be3899ace5e29f52 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.246555 15439 raft_consensus.cc:348] T ce99d91e79cf4e06b71110eada58a8e9 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.241551 15437 consensus_queue.cc:260] T ce99d91e79cf4e06b71110eada58a8e9 P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.259045 15439 raft_consensus.cc:374] T ce99d91e79cf4e06b71110eada58a8e9 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:24.259768 14695 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_6af28ee231a445709bf4b7b040fb7194"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:24.260689 15490 tablet_bootstrap.cc:492] T dc3ea02864d040c5bcde38b1a5609676 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:24.260609 14989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bbad33a92643e3be3899ace5e29f52" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224"
I20260810 08:00:24.261137 15490 ts_tablet_manager.cc:1397] T dc3ea02864d040c5bcde38b1a5609676 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.104s user 0.008s sys 0.002s
I20260810 08:00:24.261173 14989 raft_consensus.cc:3037] T 92bbad33a92643e3be3899ace5e29f52 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
W20260810 08:00:24.261862 14695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6af28ee231a445709bf4b7b040fb7194 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:24.265050 14694 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_10462271d83b4063b56a4cd0c8b6f94a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:24.266891 14694 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10462271d83b4063b56a4cd0c8b6f94a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:24.267966 14989 raft_consensus.cc:2454] T 92bbad33a92643e3be3899ace5e29f52 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 1.
I20260810 08:00:24.272104 15437 ts_tablet_manager.cc:1428] T ce99d91e79cf4e06b71110eada58a8e9 P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.034s user 0.004s sys 0.001s
I20260810 08:00:24.273100 15437 tablet_bootstrap.cc:492] T a074a741235e4c4b8db1fdb45fb971a8 P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:24.277980 15437 tablet_bootstrap.cc:654] T a074a741235e4c4b8db1fdb45fb971a8 P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:24.280014 15490 raft_consensus.cc:348] T dc3ea02864d040c5bcde38b1a5609676 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.280759 15490 raft_consensus.cc:374] T dc3ea02864d040c5bcde38b1a5609676 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:24.281432 15490 raft_consensus.cc:729] T dc3ea02864d040c5bcde38b1a5609676 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:24.282094 15490 consensus_queue.cc:260] T dc3ea02864d040c5bcde38b1a5609676 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.283946 14965 tablet_service.cc:1514] Processing DeleteTablet for tablet ce99d91e79cf4e06b71110eada58a8e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:23 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:24.284493 14690 master_service.cc:946] ReplaceTablet: received request to replace tablet ce99d91e79cf4e06b71110eada58a8e9 from {username='slave'} at 127.0.0.1:42300
I20260810 08:00:24.288591 15482 ts_tablet_manager.cc:1918] T abdd2b119a4b448dacdd2d87da34832f P a91b27762182454baada71e91ceab224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:24.288993 15482 log.cc:1196] T abdd2b119a4b448dacdd2d87da34832f P a91b27762182454baada71e91ceab224: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal/wals/abdd2b119a4b448dacdd2d87da34832f
I20260810 08:00:24.289664 15482 ts_tablet_manager.cc:1939] T abdd2b119a4b448dacdd2d87da34832f P a91b27762182454baada71e91ceab224: Deleting consensus metadata
I20260810 08:00:24.269230 15234 leader_election.cc:302] T 92bbad33a92643e3be3899ace5e29f52 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224; no voters:
I20260810 08:00:24.292078 14681 catalog_manager.cc:4841] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): tablet abdd2b119a4b448dacdd2d87da34832f (table default.Table_3a39902b550c4390a56429d0ef94860e [id=cee29bc23d044a4bae553990b0d3c32c]) successfully deleted
I20260810 08:00:24.294063 15447 leader_election.cc:288] T 92bbad33a92643e3be3899ace5e29f52 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Requested vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:24.294617 15482 tablet_replica.cc:330] T ce99d91e79cf4e06b71110eada58a8e9 P a91b27762182454baada71e91ceab224: stopping tablet replica
I20260810 08:00:24.294591 15447 raft_consensus.cc:2781] T 92bbad33a92643e3be3899ace5e29f52 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Leader election won for term 1
I20260810 08:00:24.295007 15447 raft_consensus.cc:686] T 92bbad33a92643e3be3899ace5e29f52 P 3fc5e568f3724f158b616c303efb9eae [term 1 LEADER]: Becoming Leader. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Running, Role: LEADER
I20260810 08:00:24.295315 15482 raft_consensus.cc:2229] T ce99d91e79cf4e06b71110eada58a8e9 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:24.295720 15482 raft_consensus.cc:2258] T ce99d91e79cf4e06b71110eada58a8e9 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:24.295746 15447 consensus_queue.cc:237] T 92bbad33a92643e3be3899ace5e29f52 P 3fc5e568f3724f158b616c303efb9eae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.307518 15439 raft_consensus.cc:729] T ce99d91e79cf4e06b71110eada58a8e9 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:24.308840 15482 ts_tablet_manager.cc:1905] T ce99d91e79cf4e06b71110eada58a8e9 P a91b27762182454baada71e91ceab224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:24.309356 15490 ts_tablet_manager.cc:1428] T dc3ea02864d040c5bcde38b1a5609676 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.048s user 0.007s sys 0.000s
I20260810 08:00:24.310415 15490 tablet_bootstrap.cc:492] T 5e0e4f1c76f843ea9000ed25acec73d3 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:24.312378 14690 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_6e428ff2c4c7459b84d2882a089b490d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:24.316280 15490 tablet_bootstrap.cc:654] T 5e0e4f1c76f843ea9000ed25acec73d3 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
W20260810 08:00:24.323005 14690 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e428ff2c4c7459b84d2882a089b490d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:24.331105 15490 tablet_bootstrap.cc:492] T 5e0e4f1c76f843ea9000ed25acec73d3 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:24.332404 15490 ts_tablet_manager.cc:1397] T 5e0e4f1c76f843ea9000ed25acec73d3 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.008s
I20260810 08:00:24.334913 15490 raft_consensus.cc:348] T 5e0e4f1c76f843ea9000ed25acec73d3 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.335620 15490 raft_consensus.cc:374] T 5e0e4f1c76f843ea9000ed25acec73d3 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:24.335884 15490 raft_consensus.cc:729] T 5e0e4f1c76f843ea9000ed25acec73d3 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:24.340821 14692 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_2e46207a762e4a8f9054bda89fd0293f" } new_table_name: "default.Table_6fcf472630534619ae18553e71ebedaf" modify_external_catalogs: true
I20260810 08:00:24.341876 15443 raft_consensus.cc:482] T 5e0e4f1c76f843ea9000ed25acec73d3 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:24.341190 15490 consensus_queue.cc:260] T 5e0e4f1c76f843ea9000ed25acec73d3 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.342453 15443 raft_consensus.cc:504] T 5e0e4f1c76f843ea9000ed25acec73d3 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.343863 14965 tablet_service.cc:1467] Processing CreateTablet for tablet 1461cdeda5664afd9c4c2b45199179b6 (DEFAULT_TABLE table=default.Table_d7227a2f586b4fd29babdc0ae25a0328 [id=5170908091794c4aa1d533a4a0f2b776]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:24.344612 15443 leader_election.cc:288] T 5e0e4f1c76f843ea9000ed25acec73d3 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:24.345017 14965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1461cdeda5664afd9c4c2b45199179b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:24.348567 15490 ts_tablet_manager.cc:1428] T 5e0e4f1c76f843ea9000ed25acec73d3 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.016s user 0.003s sys 0.003s
I20260810 08:00:24.349510 15490 tablet_bootstrap.cc:492] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:24.349821 15147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0e4f1c76f843ea9000ed25acec73d3" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:24.350492 15147 raft_consensus.cc:2454] T 5e0e4f1c76f843ea9000ed25acec73d3 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 0.
I20260810 08:00:24.351938 14922 leader_election.cc:302] T 5e0e4f1c76f843ea9000ed25acec73d3 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224; no voters:
I20260810 08:00:24.360190 15490 tablet_bootstrap.cc:654] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:24.360677 15443 raft_consensus.cc:2781] T 5e0e4f1c76f843ea9000ed25acec73d3 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:24.361049 15443 raft_consensus.cc:482] T 5e0e4f1c76f843ea9000ed25acec73d3 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:24.361367 15443 raft_consensus.cc:3037] T 5e0e4f1c76f843ea9000ed25acec73d3 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:24.364913 14687 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_43a063da4167472198ae9533501169eb" } modify_external_catalogs: true
I20260810 08:00:24.365594 14687 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_43a063da4167472198ae9533501169eb" } modify_external_catalogs: true
I20260810 08:00:24.373435 15490 tablet_bootstrap.cc:492] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:24.373908 15490 ts_tablet_manager.cc:1397] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.001s
I20260810 08:00:24.376389 15490 raft_consensus.cc:348] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.377151 15490 raft_consensus.cc:374] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:24.377074 15125 tablet_service.cc:1467] Processing CreateTablet for tablet 1461cdeda5664afd9c4c2b45199179b6 (DEFAULT_TABLE table=default.Table_d7227a2f586b4fd29babdc0ae25a0328 [id=5170908091794c4aa1d533a4a0f2b776]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:24.377462 15490 raft_consensus.cc:729] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:24.378273 15125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1461cdeda5664afd9c4c2b45199179b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:24.378165 15490 consensus_queue.cc:260] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.387640 15490 ts_tablet_manager.cc:1428] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20260810 08:00:24.388554 15490 tablet_bootstrap.cc:492] T 0c152aab7e244afd857534869b13815c P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:24.346235 15302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0e4f1c76f843ea9000ed25acec73d3" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:24.397905 15439 consensus_queue.cc:260] T ce99d91e79cf4e06b71110eada58a8e9 P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
W20260810 08:00:24.398895 14918 leader_election.cc:341] T 5e0e4f1c76f843ea9000ed25acec73d3 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:24.393477 15490 tablet_bootstrap.cc:654] T 0c152aab7e244afd857534869b13815c P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:24.401407 15479 tablet_replica.cc:330] T a074a741235e4c4b8db1fdb45fb971a8 P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:24.401407 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet eea24ebbcf3b4f79b8910613f1cad70c (table default.Table_5d982fdf0c924d0ebecb9de16fdebd15 [id=cd26deac07054fd6b8667aae1f1e272b]) successfully deleted
I20260810 08:00:24.402034 15482 ts_tablet_manager.cc:1918] T ce99d91e79cf4e06b71110eada58a8e9 P a91b27762182454baada71e91ceab224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:24.402395 15479 raft_consensus.cc:2229] T a074a741235e4c4b8db1fdb45fb971a8 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:24.402890 15479 raft_consensus.cc:2258] T a074a741235e4c4b8db1fdb45fb971a8 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260810 08:00:24.404877 15234 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.193:43601, user_credentials={real_user=slave}} blocked reactor thread for 135683us
I20260810 08:00:24.405856 15482 log.cc:1196] T ce99d91e79cf4e06b71110eada58a8e9 P a91b27762182454baada71e91ceab224: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal/wals/ce99d91e79cf4e06b71110eada58a8e9
I20260810 08:00:24.406807 15147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bbad33a92643e3be3899ace5e29f52" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07"
I20260810 08:00:24.407402 15147 raft_consensus.cc:3037] T 92bbad33a92643e3be3899ace5e29f52 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:24.410944 15490 tablet_bootstrap.cc:492] T 0c152aab7e244afd857534869b13815c P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:24.411363 15490 ts_tablet_manager.cc:1397] T 0c152aab7e244afd857534869b13815c P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.003s
I20260810 08:00:24.410836 14963 tablet_service.cc:1467] Processing CreateTablet for tablet 472fbc9204754317aab059a4844d7de1 (DEFAULT_TABLE table=default.Table_3630f6ea49e84b48a377bad128b8c436 [id=fc52022e20044cc3bc88ac143e088e69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:24.411960 15482 ts_tablet_manager.cc:1939] T ce99d91e79cf4e06b71110eada58a8e9 P a91b27762182454baada71e91ceab224: Deleting consensus metadata
I20260810 08:00:24.412624 14963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 472fbc9204754317aab059a4844d7de1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:24.413647 15490 raft_consensus.cc:348] T 0c152aab7e244afd857534869b13815c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.414273 15490 raft_consensus.cc:374] T 0c152aab7e244afd857534869b13815c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:24.414506 15490 raft_consensus.cc:729] T 0c152aab7e244afd857534869b13815c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:24.420222 15147 raft_consensus.cc:2454] T 92bbad33a92643e3be3899ace5e29f52 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 1.
W20260810 08:00:24.421119 15454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 08:00:24.421713 15454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 08:00:24.422176 15454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 08:00:24.422633 15454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 08:00:24.427430 15279 tablet_service.cc:1467] Processing CreateTablet for tablet 1461cdeda5664afd9c4c2b45199179b6 (DEFAULT_TABLE table=default.Table_d7227a2f586b4fd29babdc0ae25a0328 [id=5170908091794c4aa1d533a4a0f2b776]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:24.428855 15279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1461cdeda5664afd9c4c2b45199179b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:24.430296 14694 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_eb8d1ba0b9fc4c789eb3a43b70ed093b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:24.432070 14694 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb8d1ba0b9fc4c789eb3a43b70ed093b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:24.436640 15275 tablet_service.cc:1467] Processing CreateTablet for tablet 472fbc9204754317aab059a4844d7de1 (DEFAULT_TABLE table=default.Table_3630f6ea49e84b48a377bad128b8c436 [id=fc52022e20044cc3bc88ac143e088e69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:24.437826 15275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 472fbc9204754317aab059a4844d7de1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:24.441525 14695 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_3630f6ea49e84b48a377bad128b8c436" } new_table_name: "default.Table_d29065cf9a324d67a7308c95e02ad7c0" modify_external_catalogs: true
I20260810 08:00:24.450623 14687 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet 5e0e4f1c76f843ea9000ed25acec73d3
I20260810 08:00:24.459128 15276 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e0e4f1c76f843ea9000ed25acec73d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:24 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:24.420506 15490 consensus_queue.cc:260] T 0c152aab7e244afd857534869b13815c P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.461848 14967 tablet_service.cc:1467] Processing CreateTablet for tablet 9ddae63b29c34e1397c5765e52e1478f (DEFAULT_TABLE table=default.Table_0e30c621046242609c9f6fc73b16d0e7 [id=e9608d66a9964bc6aafdf84f574fba02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:24.462034 15123 tablet_service.cc:1467] Processing CreateTablet for tablet 472fbc9204754317aab059a4844d7de1 (DEFAULT_TABLE table=default.Table_3630f6ea49e84b48a377bad128b8c436 [id=fc52022e20044cc3bc88ac143e088e69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:24.462956 14967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ddae63b29c34e1397c5765e52e1478f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:24.463115 15490 ts_tablet_manager.cc:1428] T 0c152aab7e244afd857534869b13815c P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.051s user 0.005s sys 0.001s
I20260810 08:00:24.463232 15123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 472fbc9204754317aab059a4844d7de1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:24.463964 15490 tablet_bootstrap.cc:492] T b798daf216454a4598437947928803d1 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:24.468767 15122 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e0e4f1c76f843ea9000ed25acec73d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:24 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:24.469429 14966 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e0e4f1c76f843ea9000ed25acec73d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:24 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:24.470191 15482 tablet_replica.cc:330] T 5e0e4f1c76f843ea9000ed25acec73d3 P a91b27762182454baada71e91ceab224: stopping tablet replica
I20260810 08:00:24.470310 15490 tablet_bootstrap.cc:654] T b798daf216454a4598437947928803d1 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:24.470948 15443 raft_consensus.cc:504] T 5e0e4f1c76f843ea9000ed25acec73d3 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.472043 15482 raft_consensus.cc:2229] T 5e0e4f1c76f843ea9000ed25acec73d3 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:24.474826 15439 ts_tablet_manager.cc:1428] T ce99d91e79cf4e06b71110eada58a8e9 P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.230s user 0.005s sys 0.000s
I20260810 08:00:24.475790 15439 tablet_bootstrap.cc:492] T a074a741235e4c4b8db1fdb45fb971a8 P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:24.476127 15482 raft_consensus.cc:2258] T 5e0e4f1c76f843ea9000ed25acec73d3 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:24.487035 15444 raft_consensus.cc:482] T abdd2b119a4b448dacdd2d87da34832f P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:24.487528 15444 raft_consensus.cc:504] T abdd2b119a4b448dacdd2d87da34832f P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.489446 15444 leader_election.cc:288] T abdd2b119a4b448dacdd2d87da34832f P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
W20260810 08:00:24.491911 15079 leader_election.cc:341] T abdd2b119a4b448dacdd2d87da34832f P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Not found: Tablet not found: abdd2b119a4b448dacdd2d87da34832f
I20260810 08:00:24.507404 14681 catalog_manager.cc:4841] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): tablet ce99d91e79cf4e06b71110eada58a8e9 (table default.Table_82804e0d8405498b86dd27530bf9384f [id=d1c97b8884c844f8b808baaaf7038af8]) successfully deleted
I20260810 08:00:24.509186 14691 master_service.cc:946] ReplaceTablet: received request to replace tablet a074a741235e4c4b8db1fdb45fb971a8 from {username='slave'} at 127.0.0.1:42300
I20260810 08:00:24.509433 15490 tablet_bootstrap.cc:492] T b798daf216454a4598437947928803d1 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:24.509994 15490 ts_tablet_manager.cc:1397] T b798daf216454a4598437947928803d1 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.046s user 0.019s sys 0.005s
I20260810 08:00:24.511276 15122 tablet_service.cc:1467] Processing CreateTablet for tablet 9ddae63b29c34e1397c5765e52e1478f (DEFAULT_TABLE table=default.Table_0e30c621046242609c9f6fc73b16d0e7 [id=e9608d66a9964bc6aafdf84f574fba02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:24.512408 15122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ddae63b29c34e1397c5765e52e1478f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:24.513221 15490 raft_consensus.cc:348] T b798daf216454a4598437947928803d1 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.514274 15490 raft_consensus.cc:374] T b798daf216454a4598437947928803d1 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:24.514662 15490 raft_consensus.cc:729] T b798daf216454a4598437947928803d1 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:24.515497 15490 consensus_queue.cc:260] T b798daf216454a4598437947928803d1 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.526683 15479 ts_tablet_manager.cc:1905] T a074a741235e4c4b8db1fdb45fb971a8 P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:24.529521 15490 ts_tablet_manager.cc:1428] T b798daf216454a4598437947928803d1 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.019s user 0.007s sys 0.001s
I20260810 08:00:24.532238 15126 tablet_service.cc:1467] Processing CreateTablet for tablet a246b8cad99b47a9a856f6aed70bdfc8 (DEFAULT_TABLE table=default.Table_491d9caa5f6d4e3cb934a5d0582b519c [id=d106e75991f54f60abbe60b0495cb8e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:24.533969 15126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a246b8cad99b47a9a856f6aed70bdfc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:24.538769 14688 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_ac5b577421a748b09f2d5b6398a93296"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:24.540558 14688 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac5b577421a748b09f2d5b6398a93296 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:24.545151 14687 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_9b0d38b98f3841828a00a23dda57ae6e" } modify_external_catalogs: true
I20260810 08:00:24.545861 14687 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_9b0d38b98f3841828a00a23dda57ae6e" } modify_external_catalogs: true
W20260810 08:00:24.549190 14692 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 114d3eac994f466bb1787e066de05e2c has no consensus state
I20260810 08:00:24.550956 14686 catalog_manager.cc:5587] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265) does not have the latest schema for tablet aeb7c1c4bb1f45f9b46fe5bd78e96a20 (table default.Table_9b0d38b98f3841828a00a23dda57ae6e [id=40cf37d4ad57413fa64ee743a5c160c3]). Expected version 1 got 0
I20260810 08:00:24.567360 15490 tablet_bootstrap.cc:492] T 1461cdeda5664afd9c4c2b45199179b6 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:24.573024 15490 tablet_bootstrap.cc:654] T 1461cdeda5664afd9c4c2b45199179b6 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:24.574641 14692 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_1740a53bcf544546802efb49277adb22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:24.576623 14692 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1740a53bcf544546802efb49277adb22 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:24.583220 14690 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_e099089d98e946f5b61acc0673c3774f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:24.584829 14695 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 472fbc9204754317aab059a4844d7de1 consensus state has no leader
W20260810 08:00:24.585321 14690 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e099089d98e946f5b61acc0673c3774f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 08:00:24.595683 14678 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 114d3eac994f466bb1787e066de05e2c (table default.Table_6fcf472630534619ae18553e71ebedaf [id=2c633c1a27cf44f3ab04e66e2cba6e13], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 114d3eac994f466bb1787e066de05e2c has no consensus state
I20260810 08:00:24.598958 15519 raft_consensus.cc:482] T dc3ea02864d040c5bcde38b1a5609676 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:24.599450 15519 raft_consensus.cc:504] T dc3ea02864d040c5bcde38b1a5609676 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.605657 15519 leader_election.cc:288] T dc3ea02864d040c5bcde38b1a5609676 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265), a91b27762182454baada71e91ceab224 (127.14.57.193:43601)
I20260810 08:00:24.610451 14966 tablet_service.cc:1467] Processing CreateTablet for tablet a246b8cad99b47a9a856f6aed70bdfc8 (DEFAULT_TABLE table=default.Table_491d9caa5f6d4e3cb934a5d0582b519c [id=d106e75991f54f60abbe60b0495cb8e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:24.611647 14966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a246b8cad99b47a9a856f6aed70bdfc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:24.632673 15443 leader_election.cc:288] T 5e0e4f1c76f843ea9000ed25acec73d3 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Requested vote from peers 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:24.633955 15444 raft_consensus.cc:482] T ce99d91e79cf4e06b71110eada58a8e9 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:24.634500 15444 raft_consensus.cc:504] T ce99d91e79cf4e06b71110eada58a8e9 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.636518 15444 leader_election.cc:288] T ce99d91e79cf4e06b71110eada58a8e9 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:24.633579 15147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0e4f1c76f843ea9000ed25acec73d3" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07"
I20260810 08:00:24.638238 15147 raft_consensus.cc:3037] T 5e0e4f1c76f843ea9000ed25acec73d3 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:24.646585 15482 ts_tablet_manager.cc:1905] T 5e0e4f1c76f843ea9000ed25acec73d3 P a91b27762182454baada71e91ceab224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:24.648655 14687 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet aeb7c1c4bb1f45f9b46fe5bd78e96a20
I20260810 08:00:24.650091 15437 tablet_bootstrap.cc:492] T a074a741235e4c4b8db1fdb45fb971a8 P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:24.649983 15302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0e4f1c76f843ea9000ed25acec73d3" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae"
I20260810 08:00:24.650511 15437 ts_tablet_manager.cc:1397] T a074a741235e4c4b8db1fdb45fb971a8 P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.378s user 0.085s sys 0.120s
I20260810 08:00:24.653481 15437 raft_consensus.cc:348] T a074a741235e4c4b8db1fdb45fb971a8 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.654251 15437 raft_consensus.cc:374] T a074a741235e4c4b8db1fdb45fb971a8 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:24.654525 15437 raft_consensus.cc:729] T a074a741235e4c4b8db1fdb45fb971a8 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:24.655123 15437 consensus_queue.cc:260] T a074a741235e4c4b8db1fdb45fb971a8 P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.657303 15437 ts_tablet_manager.cc:1428] T a074a741235e4c4b8db1fdb45fb971a8 P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 08:00:24.658212 15437 tablet_bootstrap.cc:492] T 0c152aab7e244afd857534869b13815c P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:24.660733 15302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdd2b119a4b448dacdd2d87da34832f" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:24.662941 15276 tablet_service.cc:1467] Processing CreateTablet for tablet 9ddae63b29c34e1397c5765e52e1478f (DEFAULT_TABLE table=default.Table_0e30c621046242609c9f6fc73b16d0e7 [id=e9608d66a9964bc6aafdf84f574fba02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:24.664081 15276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ddae63b29c34e1397c5765e52e1478f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:24.664312 15437 tablet_bootstrap.cc:654] T 0c152aab7e244afd857534869b13815c P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:24.664398 15122 tablet_service.cc:1514] Processing DeleteTablet for tablet aeb7c1c4bb1f45f9b46fe5bd78e96a20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:24 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:24.676928 14687 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_491d9caa5f6d4e3cb934a5d0582b519c" } modify_external_catalogs: true
I20260810 08:00:24.677649 14687 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_491d9caa5f6d4e3cb934a5d0582b519c" } modify_external_catalogs: true
I20260810 08:00:24.682900 15274 tablet_service.cc:1467] Processing CreateTablet for tablet a246b8cad99b47a9a856f6aed70bdfc8 (DEFAULT_TABLE table=default.Table_491d9caa5f6d4e3cb934a5d0582b519c [id=d106e75991f54f60abbe60b0495cb8e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:24.684028 15274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a246b8cad99b47a9a856f6aed70bdfc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:24.685850 15278 tablet_service.cc:1514] Processing DeleteTablet for tablet aeb7c1c4bb1f45f9b46fe5bd78e96a20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:24 UTC) from {username='slave'} at 127.0.0.1:40692
W20260810 08:00:24.685767 15076 leader_election.cc:341] T abdd2b119a4b448dacdd2d87da34832f P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:24.685997 15146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3ea02864d040c5bcde38b1a5609676" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:24.686239 15076 leader_election.cc:302] T abdd2b119a4b448dacdd2d87da34832f P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:24.686664 15146 raft_consensus.cc:2454] T dc3ea02864d040c5bcde38b1a5609676 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 0.
I20260810 08:00:24.688512 15444 raft_consensus.cc:2726] T abdd2b119a4b448dacdd2d87da34832f P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:24.690016 15302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce99d91e79cf4e06b71110eada58a8e9" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:24.690630 15302 raft_consensus.cc:2454] T ce99d91e79cf4e06b71110eada58a8e9 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d7219686f304f62a54acc7497cbbd07 in term 0.
I20260810 08:00:24.692061 15076 leader_election.cc:302] T ce99d91e79cf4e06b71110eada58a8e9 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, 9d7219686f304f62a54acc7497cbbd07; no voters:
I20260810 08:00:24.692886 15444 raft_consensus.cc:2781] T ce99d91e79cf4e06b71110eada58a8e9 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:24.693221 15444 raft_consensus.cc:482] T ce99d91e79cf4e06b71110eada58a8e9 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:24.693539 15444 raft_consensus.cc:3037] T ce99d91e79cf4e06b71110eada58a8e9 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:24.693958 15454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.57.212:34009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.14.57.254:43757,127.14.57.253:40439,127.14.57.252:33819
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.57.252:33819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.57.252
--webserver_port=34789
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 07:02:49 UTC on ccycloud.quasar-ojudvd.root.comops.site
build id 4577116
TSAN enabled
I20260810 08:00:24.695649 15454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 08:00:24.698169 15454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 08:00:24.698448 15234 leader_election.cc:302] T dc3ea02864d040c5bcde38b1a5609676 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, 9d7219686f304f62a54acc7497cbbd07; no voters:
I20260810 08:00:24.688935 15482 ts_tablet_manager.cc:1918] T 5e0e4f1c76f843ea9000ed25acec73d3 P a91b27762182454baada71e91ceab224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:24.699756 15482 log.cc:1196] T 5e0e4f1c76f843ea9000ed25acec73d3 P a91b27762182454baada71e91ceab224: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal/wals/5e0e4f1c76f843ea9000ed25acec73d3
I20260810 08:00:24.700532 15482 ts_tablet_manager.cc:1939] T 5e0e4f1c76f843ea9000ed25acec73d3 P a91b27762182454baada71e91ceab224: Deleting consensus metadata
I20260810 08:00:24.701699 15444 raft_consensus.cc:504] T ce99d91e79cf4e06b71110eada58a8e9 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.746524 15519 raft_consensus.cc:2781] T dc3ea02864d040c5bcde38b1a5609676 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:24.746956 15519 raft_consensus.cc:482] T dc3ea02864d040c5bcde38b1a5609676 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:24.747303 15519 raft_consensus.cc:3037] T dc3ea02864d040c5bcde38b1a5609676 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Advancing to term 1
W20260810 08:00:24.753778 15234 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.194:36265, user_credentials={real_user=slave}} blocked reactor thread for 55411.8us
I20260810 08:00:24.755365 15519 raft_consensus.cc:504] T dc3ea02864d040c5bcde38b1a5609676 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.759133 15439 tablet_bootstrap.cc:654] T a074a741235e4c4b8db1fdb45fb971a8 P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:24.767585 14688 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_4c0b6e5e875f43e088f4f0215cf39284"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:24.769371 14688 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c0b6e5e875f43e088f4f0215cf39284 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:24.774171 14690 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_6af28ee231a445709bf4b7b040fb7194" } new_table_name: "default.Table_9b2d00d70a97483bb26390a4d1e2bb2f" modify_external_catalogs: true
I20260810 08:00:24.757510 15519 leader_election.cc:288] T dc3ea02864d040c5bcde38b1a5609676 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Requested vote from peers 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265), a91b27762182454baada71e91ceab224 (127.14.57.193:43601)
I20260810 08:00:24.728464 15478 ts_tablet_manager.cc:1918] T e0206444ca194f8f98e9d71aa858283d P 3fc5e568f3724f158b616c303efb9eae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 08:00:24.783279 15478 log.cc:1196] T e0206444ca194f8f98e9d71aa858283d P 3fc5e568f3724f158b616c303efb9eae: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal/wals/e0206444ca194f8f98e9d71aa858283d
W20260810 08:00:24.783485 15529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 08:00:24.784343 15478 ts_tablet_manager.cc:1939] T e0206444ca194f8f98e9d71aa858283d P 3fc5e568f3724f158b616c303efb9eae: Deleting consensus metadata
I20260810 08:00:24.789451 14692 catalog_manager.cc:5495] T 92bbad33a92643e3be3899ace5e29f52 P 3fc5e568f3724f158b616c303efb9eae reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fc5e568f3724f158b616c303efb9eae (127.14.57.195). New cstate: current_term: 1 leader_uuid: "3fc5e568f3724f158b616c303efb9eae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } health_report { overall_health: HEALTHY } } }
I20260810 08:00:24.799504 15146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3ea02864d040c5bcde38b1a5609676" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07"
I20260810 08:00:24.802426 15146 raft_consensus.cc:3037] T dc3ea02864d040c5bcde38b1a5609676 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Advancing to term 1
W20260810 08:00:24.805821 14918 leader_election.cc:341] T 5e0e4f1c76f843ea9000ed25acec73d3 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:24.806382 15490 tablet_bootstrap.cc:492] T 1461cdeda5664afd9c4c2b45199179b6 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:24.806900 15490 ts_tablet_manager.cc:1397] T 1461cdeda5664afd9c4c2b45199179b6 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.240s user 0.038s sys 0.062s
I20260810 08:00:24.810302 15490 raft_consensus.cc:348] T 1461cdeda5664afd9c4c2b45199179b6 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:24.811025 15490 raft_consensus.cc:374] T 1461cdeda5664afd9c4c2b45199179b6 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:24.812811 15490 raft_consensus.cc:729] T 1461cdeda5664afd9c4c2b45199179b6 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:24.813746 15490 consensus_queue.cc:260] T 1461cdeda5664afd9c4c2b45199179b6 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:24.817117 14681 catalog_manager.cc:4841] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): tablet 5e0e4f1c76f843ea9000ed25acec73d3 (table default.Table_43a063da4167472198ae9533501169eb [id=9cbd826186904ef687327cd9e02fdc1a]) successfully deleted
I20260810 08:00:24.818781 15147 raft_consensus.cc:2454] T 5e0e4f1c76f843ea9000ed25acec73d3 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 1.
I20260810 08:00:24.819392 14694 master_service.cc:946] ReplaceTablet: received request to replace tablet 5e0e4f1c76f843ea9000ed25acec73d3 from {username='slave'} at 127.0.0.1:42300
I20260810 08:00:24.820492 14922 leader_election.cc:302] T 5e0e4f1c76f843ea9000ed25acec73d3 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224; no voters: 3fc5e568f3724f158b616c303efb9eae
W20260810 08:00:24.821360 14922 raft_consensus.cc:2670] T 5e0e4f1c76f843ea9000ed25acec73d3 P a91b27762182454baada71e91ceab224: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260810 08:00:24.833552 14695 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_d29065cf9a324d67a7308c95e02ad7c0" } new_table_name: "default.Table_eab0fdadaf49486ca724a0a6e179a889" modify_external_catalogs: true
I20260810 08:00:24.838717 15439 tablet_bootstrap.cc:492] T a074a741235e4c4b8db1fdb45fb971a8 P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:24.839143 15439 ts_tablet_manager.cc:1397] T a074a741235e4c4b8db1fdb45fb971a8 P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.364s user 0.007s sys 0.002s
I20260810 08:00:24.839087 14988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3ea02864d040c5bcde38b1a5609676" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:24.848421 15437 tablet_bootstrap.cc:492] T 0c152aab7e244afd857534869b13815c P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:24.842504 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3ea02864d040c5bcde38b1a5609676" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224"
I20260810 08:00:24.845444 14691 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_dbbc4dceb17d4aecb41b3ec5fc8c21ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:24.851608 14691 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dbbc4dceb17d4aecb41b3ec5fc8c21ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:24.852510 14678 catalog_manager.cc:4841] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): tablet e0206444ca194f8f98e9d71aa858283d (table default.Table_0e30c621046242609c9f6fc73b16d0e7 [id=e9608d66a9964bc6aafdf84f574fba02]) successfully deleted
I20260810 08:00:24.849123 15437 ts_tablet_manager.cc:1397] T 0c152aab7e244afd857534869b13815c P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.191s user 0.023s sys 0.028s
I20260810 08:00:24.857573 14966 tablet_service.cc:1514] Processing DeleteTablet for tablet aeb7c1c4bb1f45f9b46fe5bd78e96a20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:24 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:24.860241 15437 raft_consensus.cc:348] T 0c152aab7e244afd857534869b13815c P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
W20260810 08:00:24.863131 14681 catalog_manager.cc:4821] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet aeb7c1c4bb1f45f9b46fe5bd78e96a20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aeb7c1c4bb1f45f9b46fe5bd78e96a20 already in progress: creating tablet
I20260810 08:00:24.861430 15437 raft_consensus.cc:374] T 0c152aab7e244afd857534869b13815c P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:24.864054 15437 raft_consensus.cc:729] T 0c152aab7e244afd857534869b13815c P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:24.864698 15437 consensus_queue.cc:260] T 0c152aab7e244afd857534869b13815c P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
W20260810 08:00:24.866310 15234 leader_election.cc:341] T dc3ea02864d040c5bcde38b1a5609676 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
W20260810 08:00:24.867276 15234 leader_election.cc:341] T dc3ea02864d040c5bcde38b1a5609676 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:24.870663 15443 raft_consensus.cc:482] T a074a741235e4c4b8db1fdb45fb971a8 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:24.871099 15443 raft_consensus.cc:504] T a074a741235e4c4b8db1fdb45fb971a8 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.876206 15478 tablet_replica.cc:330] T ce99d91e79cf4e06b71110eada58a8e9 P 3fc5e568f3724f158b616c303efb9eae: stopping tablet replica
I20260810 08:00:24.877040 15478 raft_consensus.cc:2229] T ce99d91e79cf4e06b71110eada58a8e9 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:24.877507 15478 raft_consensus.cc:2258] T ce99d91e79cf4e06b71110eada58a8e9 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:24.877535 14694 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_10462271d83b4063b56a4cd0c8b6f94a" } modify_external_catalogs: true
I20260810 08:00:24.878397 14694 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_10462271d83b4063b56a4cd0c8b6f94a" } modify_external_catalogs: true
I20260810 08:00:24.879338 15478 ts_tablet_manager.cc:1905] T ce99d91e79cf4e06b71110eada58a8e9 P 3fc5e568f3724f158b616c303efb9eae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:24.881995 15439 raft_consensus.cc:348] T a074a741235e4c4b8db1fdb45fb971a8 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
W20260810 08:00:24.876298 14690 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5a22a48e2ba843b3a32a8900ab937ea0 consensus state has no leader
I20260810 08:00:24.882908 15439 raft_consensus.cc:374] T a074a741235e4c4b8db1fdb45fb971a8 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:24.884523 15437 ts_tablet_manager.cc:1428] T 0c152aab7e244afd857534869b13815c P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.028s user 0.008s sys 0.000s
I20260810 08:00:24.884343 15443 leader_election.cc:288] T a074a741235e4c4b8db1fdb45fb971a8 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:24.890395 15302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a074a741235e4c4b8db1fdb45fb971a8" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:24.890806 14966 tablet_service.cc:1514] Processing DeleteTablet for tablet a074a741235e4c4b8db1fdb45fb971a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:23 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:24.891808 15482 tablet_replica.cc:330] T a074a741235e4c4b8db1fdb45fb971a8 P a91b27762182454baada71e91ceab224: stopping tablet replica
I20260810 08:00:24.892632 15482 raft_consensus.cc:2229] T a074a741235e4c4b8db1fdb45fb971a8 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:24.893172 15482 raft_consensus.cc:2258] T a074a741235e4c4b8db1fdb45fb971a8 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:24.900233 15437 tablet_bootstrap.cc:492] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:24.900530 15482 ts_tablet_manager.cc:1905] T a074a741235e4c4b8db1fdb45fb971a8 P a91b27762182454baada71e91ceab224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:24.901315 14687 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet a246b8cad99b47a9a856f6aed70bdfc8
I20260810 08:00:24.883164 15439 raft_consensus.cc:729] T a074a741235e4c4b8db1fdb45fb971a8 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:24.902868 14692 master_service.cc:946] ReplaceTablet: received request to replace tablet 92bbad33a92643e3be3899ace5e29f52 from {username='slave'} at 127.0.0.1:42300
I20260810 08:00:24.903787 15276 tablet_service.cc:1514] Processing DeleteTablet for tablet a246b8cad99b47a9a856f6aed70bdfc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:24 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:24.906950 14966 tablet_service.cc:1514] Processing DeleteTablet for tablet a246b8cad99b47a9a856f6aed70bdfc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:24 UTC) from {username='slave'} at 127.0.0.1:34630
W20260810 08:00:24.908516 15079 leader_election.cc:341] T ce99d91e79cf4e06b71110eada58a8e9 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Not found: Tablet not found: ce99d91e79cf4e06b71110eada58a8e9
W20260810 08:00:24.909168 15079 leader_election.cc:341] T ce99d91e79cf4e06b71110eada58a8e9 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Not found: Tablet not found: ce99d91e79cf4e06b71110eada58a8e9
I20260810 08:00:24.904650 15439 consensus_queue.cc:260] T a074a741235e4c4b8db1fdb45fb971a8 P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.909893 15302 raft_consensus.cc:2454] T a074a741235e4c4b8db1fdb45fb971a8 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 0.
I20260810 08:00:24.912271 15147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a074a741235e4c4b8db1fdb45fb971a8" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:24.913785 15302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce99d91e79cf4e06b71110eada58a8e9" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae"
I20260810 08:00:24.915195 14918 leader_election.cc:302] T a074a741235e4c4b8db1fdb45fb971a8 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224; no voters:
I20260810 08:00:24.915403 14688 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_da20d9f3b8384f3fac7f39669323006b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:24.916122 14918 raft_consensus.cc:2670] T a074a741235e4c4b8db1fdb45fb971a8 P a91b27762182454baada71e91ceab224: unable to run election callback: Service unavailable: Thread pool token was shut down
W20260810 08:00:24.918056 14688 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da20d9f3b8384f3fac7f39669323006b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:24.924017 15439 ts_tablet_manager.cc:1428] T a074a741235e4c4b8db1fdb45fb971a8 P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.085s user 0.007s sys 0.000s
I20260810 08:00:24.926294 15439 tablet_bootstrap.cc:492] T 5e0e4f1c76f843ea9000ed25acec73d3 P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
W20260810 08:00:24.942371 14695 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 472fbc9204754317aab059a4844d7de1 consensus state has no leader
W20260810 08:00:24.944947 15076 leader_election.cc:341] T ce99d91e79cf4e06b71110eada58a8e9 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: Tablet not RUNNING: STOPPED
I20260810 08:00:24.913357 15126 tablet_service.cc:1514] Processing DeleteTablet for tablet a246b8cad99b47a9a856f6aed70bdfc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:24 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:24.949889 15146 raft_consensus.cc:2454] T dc3ea02864d040c5bcde38b1a5609676 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 1.
I20260810 08:00:24.945549 15076 leader_election.cc:302] T ce99d91e79cf4e06b71110eada58a8e9 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:24.951098 15234 leader_election.cc:302] T dc3ea02864d040c5bcde38b1a5609676 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, 9d7219686f304f62a54acc7497cbbd07; no voters: a91b27762182454baada71e91ceab224
I20260810 08:00:24.952281 15519 raft_consensus.cc:2781] T dc3ea02864d040c5bcde38b1a5609676 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Leader election won for term 1
I20260810 08:00:24.952754 15519 raft_consensus.cc:686] T dc3ea02864d040c5bcde38b1a5609676 P 3fc5e568f3724f158b616c303efb9eae [term 1 LEADER]: Becoming Leader. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Running, Role: LEADER
I20260810 08:00:24.954281 15479 ts_tablet_manager.cc:1918] T a074a741235e4c4b8db1fdb45fb971a8 P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:24.954660 15479 log.cc:1196] T a074a741235e4c4b8db1fdb45fb971a8 P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/a074a741235e4c4b8db1fdb45fb971a8
I20260810 08:00:24.955302 15479 ts_tablet_manager.cc:1939] T a074a741235e4c4b8db1fdb45fb971a8 P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:24.953526 15519 consensus_queue.cc:237] T dc3ea02864d040c5bcde38b1a5609676 P 3fc5e568f3724f158b616c303efb9eae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.956066 15479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43002 (request call id 15) took 1043 ms. Trace:
I20260810 08:00:24.956399 15479 rpcz_store.cc:276] 0810 08:00:23.912610 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:23.912766 (+ 156us) service_pool.cc:227] Handling call
0810 08:00:24.401227 (+488461us) ts_tablet_manager.cc:1145] Deleting tablet a074a741235e4c4b8db1fdb45fb971a8
0810 08:00:24.401275 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:24.911499 (+510224us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:24.954040 (+ 42541us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:24.956012 (+ 1972us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":487857}
I20260810 08:00:24.958038 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet a074a741235e4c4b8db1fdb45fb971a8 (table default.Table_dcbe00e111e246a3975b5a4bd8cce1a0 [id=72cc95ab8de44814aaeece045b4d8d77]) successfully deleted
I20260810 08:00:24.960356 14690 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_233aa3d7e25f4dae8cbf9e3303c22cf1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:24.964560 15437 tablet_bootstrap.cc:654] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
W20260810 08:00:24.968827 14922 leader_election.cc:341] T a074a741235e4c4b8db1fdb45fb971a8 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Illegal state: Tablet not RUNNING: STOPPED
W20260810 08:00:24.968909 14690 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_233aa3d7e25f4dae8cbf9e3303c22cf1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 08:00:24.969388 14681 consensus_peers.cc:487] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 -> Peer 1e8cadb26fe0428a964f66ef8172f3be (127.14.57.252:33819): Couldn't send request to peer 1e8cadb26fe0428a964f66ef8172f3be. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.252:33819: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260810 08:00:24.981108 15538 raft_consensus.cc:482] T 0c152aab7e244afd857534869b13815c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:24.985888 15539 raft_consensus.cc:2726] T ce99d91e79cf4e06b71110eada58a8e9 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 08:00:24.983174 15538 raft_consensus.cc:504] T 0c152aab7e244afd857534869b13815c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.990995 15539 raft_consensus.cc:482] T b798daf216454a4598437947928803d1 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:24.991464 15539 raft_consensus.cc:504] T b798daf216454a4598437947928803d1 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:24.995076 15538 leader_election.cc:288] T 0c152aab7e244afd857534869b13815c P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:24.993541 15539 leader_election.cc:288] T b798daf216454a4598437947928803d1 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:24.995275 14989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b798daf216454a4598437947928803d1" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:24.943107 15444 leader_election.cc:288] T ce99d91e79cf4e06b71110eada58a8e9 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 election: Requested vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:24.998710 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c152aab7e244afd857534869b13815c" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:24.999298 14987 raft_consensus.cc:2454] T 0c152aab7e244afd857534869b13815c P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d7219686f304f62a54acc7497cbbd07 in term 0.
W20260810 08:00:25.000245 15079 leader_election.cc:341] T b798daf216454a4598437947928803d1 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:25.000922 15079 leader_election.cc:302] T 0c152aab7e244afd857534869b13815c P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224; no voters:
W20260810 08:00:25.001468 15076 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.195:37729, user_credentials={real_user=slave}} blocked reactor thread for 56574.5us
I20260810 08:00:25.001727 15444 raft_consensus.cc:2781] T 0c152aab7e244afd857534869b13815c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:25.002050 15444 raft_consensus.cc:482] T 0c152aab7e244afd857534869b13815c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:25.002362 15444 raft_consensus.cc:3037] T 0c152aab7e244afd857534869b13815c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:25.003528 15302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b798daf216454a4598437947928803d1" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:25.004786 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c152aab7e244afd857534869b13815c" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:25.007637 15126 tablet_service.cc:1467] Processing CreateTablet for tablet 2faab63c5023431c98e7b01ce61e04c2 (DEFAULT_TABLE table=default.Table_10462271d83b4063b56a4cd0c8b6f94a [id=67e4dae7d49b48669b176960841b25ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:25.008343 15276 tablet_service.cc:1467] Processing CreateTablet for tablet 2faab63c5023431c98e7b01ce61e04c2 (DEFAULT_TABLE table=default.Table_10462271d83b4063b56a4cd0c8b6f94a [id=67e4dae7d49b48669b176960841b25ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:25.008796 15126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2faab63c5023431c98e7b01ce61e04c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:25.009455 15276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2faab63c5023431c98e7b01ce61e04c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:25.014000 15479 tablet_replica.cc:330] T e0206444ca194f8f98e9d71aa858283d P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:25.014597 14963 tablet_service.cc:1467] Processing CreateTablet for tablet 35b6da60a5544d7e9b657a5f1e6f90ff (DEFAULT_TABLE table=default.Table_6e428ff2c4c7459b84d2882a089b490d [id=64f8570d723d4f5faba292689c535c01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:25.014855 15479 raft_consensus.cc:2229] T e0206444ca194f8f98e9d71aa858283d P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:25.015331 15479 raft_consensus.cc:2258] T e0206444ca194f8f98e9d71aa858283d P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:25.015832 14963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35b6da60a5544d7e9b657a5f1e6f90ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:25.017135 15479 ts_tablet_manager.cc:1905] T e0206444ca194f8f98e9d71aa858283d P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:25.019619 14966 tablet_service.cc:1467] Processing CreateTablet for tablet 2faab63c5023431c98e7b01ce61e04c2 (DEFAULT_TABLE table=default.Table_10462271d83b4063b56a4cd0c8b6f94a [id=67e4dae7d49b48669b176960841b25ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:25.020737 14966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2faab63c5023431c98e7b01ce61e04c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:25.024266 15490 ts_tablet_manager.cc:1428] T 1461cdeda5664afd9c4c2b45199179b6 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.217s user 0.006s sys 0.002s
I20260810 08:00:25.025251 15490 tablet_bootstrap.cc:492] T 9ddae63b29c34e1397c5765e52e1478f P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:25.035283 15490 tablet_bootstrap.cc:654] T 9ddae63b29c34e1397c5765e52e1478f P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:25.041131 15279 tablet_service.cc:1467] Processing CreateTablet for tablet 35b6da60a5544d7e9b657a5f1e6f90ff (DEFAULT_TABLE table=default.Table_6e428ff2c4c7459b84d2882a089b490d [id=64f8570d723d4f5faba292689c535c01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:25.042382 15279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35b6da60a5544d7e9b657a5f1e6f90ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:25.042235 15123 tablet_service.cc:1467] Processing CreateTablet for tablet 35b6da60a5544d7e9b657a5f1e6f90ff (DEFAULT_TABLE table=default.Table_6e428ff2c4c7459b84d2882a089b490d [id=64f8570d723d4f5faba292689c535c01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:25.043531 15123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35b6da60a5544d7e9b657a5f1e6f90ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:25.044726 15444 raft_consensus.cc:504] T 0c152aab7e244afd857534869b13815c P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:25.051201 15444 leader_election.cc:288] T 0c152aab7e244afd857534869b13815c P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 election: Requested vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:25.052659 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c152aab7e244afd857534869b13815c" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224"
I20260810 08:00:25.053249 14987 raft_consensus.cc:3037] T 0c152aab7e244afd857534869b13815c P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:25.056140 15437 tablet_bootstrap.cc:492] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:25.056738 15437 ts_tablet_manager.cc:1397] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.157s user 0.012s sys 0.010s
I20260810 08:00:25.066409 15439 tablet_bootstrap.cc:700] T 5e0e4f1c76f843ea9000ed25acec73d3 P 3fc5e568f3724f158b616c303efb9eae: Time spent opening tablet: real 0.116s user 0.001s sys 0.000s
I20260810 08:00:25.067147 15439 tablet_bootstrap.cc:654] T 5e0e4f1c76f843ea9000ed25acec73d3 P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:25.074255 14692 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet 92bbad33a92643e3be3899ace5e29f52
I20260810 08:00:25.076169 14965 tablet_service.cc:1514] Processing DeleteTablet for tablet 92bbad33a92643e3be3899ace5e29f52 with delete_type TABLET_DATA_DELETED (replaced by tablet fbf62f4eab8b49f282df3ab4d95d886f) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:25.077983 14692 catalog_manager.cc:6565] ReplaceTablet: tablet 92bbad33a92643e3be3899ace5e29f52 deleted and replaced by tablet fbf62f4eab8b49f282df3ab4d95d886f
I20260810 08:00:25.078518 15275 tablet_service.cc:1514] Processing DeleteTablet for tablet 92bbad33a92643e3be3899ace5e29f52 with delete_type TABLET_DATA_DELETED (replaced by tablet fbf62f4eab8b49f282df3ab4d95d886f) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:25.082868 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c152aab7e244afd857534869b13815c" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae"
W20260810 08:00:25.084481 15076 leader_election.cc:341] T b798daf216454a4598437947928803d1 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:25.085983 15126 tablet_service.cc:1514] Processing DeleteTablet for tablet 92bbad33a92643e3be3899ace5e29f52 with delete_type TABLET_DATA_DELETED (replaced by tablet fbf62f4eab8b49f282df3ab4d95d886f) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:25.085006 15076 leader_election.cc:302] T b798daf216454a4598437947928803d1 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:25.086824 15444 raft_consensus.cc:2726] T b798daf216454a4598437947928803d1 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 08:00:25.087287 15076 leader_election.cc:341] T 0c152aab7e244afd857534869b13815c P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:25.088222 14987 raft_consensus.cc:2454] T 0c152aab7e244afd857534869b13815c P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d7219686f304f62a54acc7497cbbd07 in term 1.
W20260810 08:00:25.088685 15076 leader_election.cc:341] T 0c152aab7e244afd857534869b13815c P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:25.092953 14694 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet 2faab63c5023431c98e7b01ce61e04c2
I20260810 08:00:25.094962 14965 tablet_service.cc:1514] Processing DeleteTablet for tablet 2faab63c5023431c98e7b01ce61e04c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:24 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:25.096454 15275 tablet_service.cc:1514] Processing DeleteTablet for tablet 2faab63c5023431c98e7b01ce61e04c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:24 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:25.098383 14691 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_1740a53bcf544546802efb49277adb22" } new_table_name: "default.Table_bae5256f457944049d86e384fb3759d6" modify_external_catalogs: true
I20260810 08:00:25.104139 14965 tablet_service.cc:1467] Processing CreateTablet for tablet 114d3eac994f466bb1787e066de05e2c (DEFAULT_TABLE table=default.Table_6fcf472630534619ae18553e71ebedaf [id=2c633c1a27cf44f3ab04e66e2cba6e13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:25.105440 14965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 114d3eac994f466bb1787e066de05e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:25.115118 15437 raft_consensus.cc:348] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:25.116039 15437 raft_consensus.cc:374] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:25.116376 15437 raft_consensus.cc:729] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:25.117219 15437 consensus_queue.cc:260] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:25.119225 14688 catalog_manager.cc:5495] T dc3ea02864d040c5bcde38b1a5609676 P 3fc5e568f3724f158b616c303efb9eae reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fc5e568f3724f158b616c303efb9eae (127.14.57.195). New cstate: current_term: 1 leader_uuid: "3fc5e568f3724f158b616c303efb9eae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } health_report { overall_health: HEALTHY } } }
I20260810 08:00:25.132691 14690 master_service.cc:946] ReplaceTablet: received request to replace tablet a246b8cad99b47a9a856f6aed70bdfc8 from {username='slave'} at 127.0.0.1:42300
W20260810 08:00:25.133315 15076 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.195:37729, user_credentials={real_user=slave}} blocked reactor thread for 44683.8us
I20260810 08:00:25.138242 14694 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_ac5b577421a748b09f2d5b6398a93296" } modify_external_catalogs: true
I20260810 08:00:25.138871 14694 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_ac5b577421a748b09f2d5b6398a93296" } modify_external_catalogs: true
I20260810 08:00:25.136911 15079 leader_election.cc:302] T 0c152aab7e244afd857534869b13815c P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224; no voters: 3fc5e568f3724f158b616c303efb9eae
I20260810 08:00:25.142483 15444 raft_consensus.cc:2781] T 0c152aab7e244afd857534869b13815c P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader election won for term 1
W20260810 08:00:25.142597 15079 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.193:43601, user_credentials={real_user=slave}} blocked reactor thread for 53068.7us
I20260810 08:00:25.143030 15444 raft_consensus.cc:686] T 0c152aab7e244afd857534869b13815c P 9d7219686f304f62a54acc7497cbbd07 [term 1 LEADER]: Becoming Leader. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Running, Role: LEADER
I20260810 08:00:25.143985 14686 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_28276fa7c9a042209ceaf8cc7c13eb43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:25.143944 15444 consensus_queue.cc:237] T 0c152aab7e244afd857534869b13815c P 9d7219686f304f62a54acc7497cbbd07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
W20260810 08:00:25.146099 14686 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28276fa7c9a042209ceaf8cc7c13eb43 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:25.156468 15275 tablet_service.cc:1467] Processing CreateTablet for tablet 114d3eac994f466bb1787e066de05e2c (DEFAULT_TABLE table=default.Table_6fcf472630534619ae18553e71ebedaf [id=2c633c1a27cf44f3ab04e66e2cba6e13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:25.157832 15275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 114d3eac994f466bb1787e066de05e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:25.163851 15274 tablet_service.cc:1467] Processing CreateTablet for tablet a1c4f8f514c94128aedd07bd5efdc3be (DEFAULT_TABLE table=default.Table_eb8d1ba0b9fc4c789eb3a43b70ed093b [id=ce96e7fb7c46404a859ca08c1fd1b26a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:25.167799 15274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1c4f8f514c94128aedd07bd5efdc3be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:25.181193 14967 tablet_service.cc:1467] Processing CreateTablet for tablet a1c4f8f514c94128aedd07bd5efdc3be (DEFAULT_TABLE table=default.Table_eb8d1ba0b9fc4c789eb3a43b70ed093b [id=ce96e7fb7c46404a859ca08c1fd1b26a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:25.182410 14967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1c4f8f514c94128aedd07bd5efdc3be. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 08:00:25.168814 14691 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 37d8a37a16f34a0285c43c0d71b5f072 has no consensus state
I20260810 08:00:25.180078 14695 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_e099089d98e946f5b61acc0673c3774f" } new_table_name: "default.Table_b017ccb238ff4e75afbdf87098ee3bcc" modify_external_catalogs: true
I20260810 08:00:25.206122 15439 tablet_bootstrap.cc:492] T 5e0e4f1c76f843ea9000ed25acec73d3 P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:25.206558 15439 ts_tablet_manager.cc:1397] T 5e0e4f1c76f843ea9000ed25acec73d3 P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.280s user 0.007s sys 0.003s
I20260810 08:00:25.208333 15126 tablet_service.cc:1514] Processing DeleteTablet for tablet 2faab63c5023431c98e7b01ce61e04c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:24 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:25.210665 15126 tablet_service.cc:1467] Processing CreateTablet for tablet 114d3eac994f466bb1787e066de05e2c (DEFAULT_TABLE table=default.Table_6fcf472630534619ae18553e71ebedaf [id=2c633c1a27cf44f3ab04e66e2cba6e13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:25.211897 15126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 114d3eac994f466bb1787e066de05e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:25.211972 15437 ts_tablet_manager.cc:1428] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.155s user 0.006s sys 0.000s
I20260810 08:00:25.192512 14692 catalog_manager.cc:5495] T 0c152aab7e244afd857534869b13815c P 9d7219686f304f62a54acc7497cbbd07 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194). New cstate: current_term: 1 leader_uuid: "9d7219686f304f62a54acc7497cbbd07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } health_report { overall_health: UNKNOWN } } }
I20260810 08:00:25.222450 15444 raft_consensus.cc:482] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:25.222945 15444 raft_consensus.cc:504] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:25.224831 15444 leader_election.cc:288] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:25.226799 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeb7c1c4bb1f45f9b46fe5bd78e96a20" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:25.227689 14987 raft_consensus.cc:2454] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d7219686f304f62a54acc7497cbbd07 in term 0.
I20260810 08:00:25.229285 15079 leader_election.cc:302] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224; no voters:
I20260810 08:00:25.230131 15444 raft_consensus.cc:2781] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260810 08:00:25.230199 14695 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 49c35b7f4a72437fa0675abe8a8f3a71 has no consensus state
I20260810 08:00:25.230469 15444 raft_consensus.cc:482] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:25.230825 15444 raft_consensus.cc:3037] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:25.239367 14964 tablet_service.cc:1467] Processing CreateTablet for tablet 5a22a48e2ba843b3a32a8900ab937ea0 (DEFAULT_TABLE table=default.Table_9b2d00d70a97483bb26390a4d1e2bb2f [id=ac3a5787385940bca8030d7bac893a27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:25.240685 14964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a22a48e2ba843b3a32a8900ab937ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:25.246223 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeb7c1c4bb1f45f9b46fe5bd78e96a20" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
W20260810 08:00:25.249541 15076 leader_election.cc:341] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:25.250782 14962 tablet_service.cc:1514] Processing DeleteTablet for tablet aeb7c1c4bb1f45f9b46fe5bd78e96a20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:24 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:25.251435 14695 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_a6ce85a21a354e7c92f158c1ea3212b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:25.254273 14695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a6ce85a21a354e7c92f158c1ea3212b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:25.226151 15519 consensus_queue.cc:1035] T 92bbad33a92643e3be3899ace5e29f52 P 3fc5e568f3724f158b616c303efb9eae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.031s
I20260810 08:00:25.244637 15482 ts_tablet_manager.cc:1918] T a074a741235e4c4b8db1fdb45fb971a8 P a91b27762182454baada71e91ceab224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:25.270778 15482 log.cc:1196] T a074a741235e4c4b8db1fdb45fb971a8 P a91b27762182454baada71e91ceab224: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal/wals/a074a741235e4c4b8db1fdb45fb971a8
I20260810 08:00:25.272589 15444 raft_consensus.cc:504] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:25.276072 14691 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_5e1d024b54504287840e8c45c056fa56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:25.276466 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeb7c1c4bb1f45f9b46fe5bd78e96a20" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae"
W20260810 08:00:25.277675 15076 leader_election.cc:341] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:25.278446 15444 leader_election.cc:288] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 election: Requested vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
W20260810 08:00:25.279072 14691 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e1d024b54504287840e8c45c056fa56 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:25.290374 14694 catalog_manager.cc:5777] Not sending DeleteTablet requests; no consensus state for tablet 4a17ceee048b49ec9dc2b1ab46d920f3
I20260810 08:00:25.291301 15437 tablet_bootstrap.cc:492] T dc3ea02864d040c5bcde38b1a5609676 P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:25.297966 14692 master_service.cc:946] ReplaceTablet: received request to replace tablet dc3ea02864d040c5bcde38b1a5609676 from {username='slave'} at 127.0.0.1:42300
I20260810 08:00:25.301757 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeb7c1c4bb1f45f9b46fe5bd78e96a20" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224"
I20260810 08:00:25.302444 14987 raft_consensus.cc:3037] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:25.302822 15123 tablet_service.cc:1467] Processing CreateTablet for tablet a1c4f8f514c94128aedd07bd5efdc3be (DEFAULT_TABLE table=default.Table_eb8d1ba0b9fc4c789eb3a43b70ed093b [id=ce96e7fb7c46404a859ca08c1fd1b26a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:25.302626 15439 raft_consensus.cc:348] T 5e0e4f1c76f843ea9000ed25acec73d3 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:25.303336 15439 raft_consensus.cc:374] T 5e0e4f1c76f843ea9000ed25acec73d3 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:25.303272 15126 tablet_service.cc:1467] Processing CreateTablet for tablet 5a22a48e2ba843b3a32a8900ab937ea0 (DEFAULT_TABLE table=default.Table_9b2d00d70a97483bb26390a4d1e2bb2f [id=ac3a5787385940bca8030d7bac893a27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:25.303579 15439 raft_consensus.cc:729] T 5e0e4f1c76f843ea9000ed25acec73d3 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:25.303831 15123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1c4f8f514c94128aedd07bd5efdc3be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:25.304337 15126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a22a48e2ba843b3a32a8900ab937ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:25.304276 15439 consensus_queue.cc:260] T 5e0e4f1c76f843ea9000ed25acec73d3 P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:25.310070 15439 ts_tablet_manager.cc:1428] T 5e0e4f1c76f843ea9000ed25acec73d3 P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.103s user 0.005s sys 0.001s
I20260810 08:00:25.310084 14688 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_6fcf472630534619ae18553e71ebedaf" } modify_external_catalogs: true
I20260810 08:00:25.310645 14688 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_6fcf472630534619ae18553e71ebedaf" } modify_external_catalogs: true
I20260810 08:00:25.311113 15439 tablet_bootstrap.cc:492] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:25.313874 15279 tablet_service.cc:1514] Processing DeleteTablet for tablet a074a741235e4c4b8db1fdb45fb971a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:23 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:25.314448 15478 ts_tablet_manager.cc:1918] T ce99d91e79cf4e06b71110eada58a8e9 P 3fc5e568f3724f158b616c303efb9eae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:25.314915 15478 log.cc:1196] T ce99d91e79cf4e06b71110eada58a8e9 P 3fc5e568f3724f158b616c303efb9eae: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal/wals/ce99d91e79cf4e06b71110eada58a8e9
I20260810 08:00:25.315806 15478 ts_tablet_manager.cc:1939] T ce99d91e79cf4e06b71110eada58a8e9 P 3fc5e568f3724f158b616c303efb9eae: Deleting consensus metadata
I20260810 08:00:25.318110 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 20) took 1180 ms. Trace:
I20260810 08:00:25.334635 15490 tablet_bootstrap.cc:492] T 9ddae63b29c34e1397c5765e52e1478f P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:25.335209 15490 ts_tablet_manager.cc:1397] T 9ddae63b29c34e1397c5765e52e1478f P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.310s user 0.009s sys 0.002s
I20260810 08:00:25.338286 14686 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_1e20f010e84746b0a487c82539f9ab8a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:25.340284 14686 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e20f010e84746b0a487c82539f9ab8a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:25.351853 15482 ts_tablet_manager.cc:1939] T a074a741235e4c4b8db1fdb45fb971a8 P a91b27762182454baada71e91ceab224: Deleting consensus metadata
I20260810 08:00:25.355290 14681 catalog_manager.cc:4841] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): tablet a074a741235e4c4b8db1fdb45fb971a8 (table default.Table_dcbe00e111e246a3975b5a4bd8cce1a0 [id=72cc95ab8de44814aaeece045b4d8d77]) successfully deleted
W20260810 08:00:25.363174 14681 catalog_manager.cc:4821] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet a246b8cad99b47a9a856f6aed70bdfc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a246b8cad99b47a9a856f6aed70bdfc8 already in progress: creating tablet
I20260810 08:00:25.370750 15490 raft_consensus.cc:348] T 9ddae63b29c34e1397c5765e52e1478f P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:25.371685 15490 raft_consensus.cc:374] T 9ddae63b29c34e1397c5765e52e1478f P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:25.372114 15490 raft_consensus.cc:729] T 9ddae63b29c34e1397c5765e52e1478f P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:25.373056 15490 consensus_queue.cc:260] T 9ddae63b29c34e1397c5765e52e1478f P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:25.390841 14987 raft_consensus.cc:2454] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d7219686f304f62a54acc7497cbbd07 in term 1.
I20260810 08:00:25.403501 14687 master_service.cc:946] ReplaceTablet: received request to replace tablet 114d3eac994f466bb1787e066de05e2c from {username='slave'} at 127.0.0.1:42300
I20260810 08:00:25.297178 15276 tablet_service.cc:1467] Processing CreateTablet for tablet 5a22a48e2ba843b3a32a8900ab937ea0 (DEFAULT_TABLE table=default.Table_9b2d00d70a97483bb26390a4d1e2bb2f [id=ac3a5787385940bca8030d7bac893a27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:25.410526 15276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a22a48e2ba843b3a32a8900ab937ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:25.409765 15478 rpcz_store.cc:276] 0810 08:00:24.137281 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:24.137422 (+ 141us) service_pool.cc:227] Handling call
0810 08:00:24.875978 (+738556us) ts_tablet_manager.cc:1145] Deleting tablet ce99d91e79cf4e06b71110eada58a8e9
0810 08:00:24.876027 (+ 49us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:25.065218 (+189191us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:25.125341 (+ 60123us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:25.318046 (+192705us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":737885}
I20260810 08:00:25.411818 14678 catalog_manager.cc:4841] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): tablet ce99d91e79cf4e06b71110eada58a8e9 (table default.Table_82804e0d8405498b86dd27530bf9384f [id=d1c97b8884c844f8b808baaaf7038af8]) successfully deleted
I20260810 08:00:25.413743 15079 leader_election.cc:302] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224; no voters: 3fc5e568f3724f158b616c303efb9eae
I20260810 08:00:25.415390 15444 raft_consensus.cc:2781] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 08:00:25.415860 15444 raft_consensus.cc:686] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 9d7219686f304f62a54acc7497cbbd07 [term 1 LEADER]: Becoming Leader. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Running, Role: LEADER
I20260810 08:00:25.416656 15444 consensus_queue.cc:237] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 9d7219686f304f62a54acc7497cbbd07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:25.359262 15519 consensus_queue.cc:1035] T 92bbad33a92643e3be3899ace5e29f52 P 3fc5e568f3724f158b616c303efb9eae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 08:00:25.422806 15482 tablet_replica.cc:330] T 92bbad33a92643e3be3899ace5e29f52 P a91b27762182454baada71e91ceab224: stopping tablet replica
I20260810 08:00:25.423748 15482 raft_consensus.cc:2229] T 92bbad33a92643e3be3899ace5e29f52 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:25.424299 15482 raft_consensus.cc:2258] T 92bbad33a92643e3be3899ace5e29f52 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:25.426147 15439 tablet_bootstrap.cc:654] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:25.427389 15482 ts_tablet_manager.cc:1905] T 92bbad33a92643e3be3899ace5e29f52 P a91b27762182454baada71e91ceab224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:25.428215 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 22) took 1207 ms. Trace:
I20260810 08:00:25.428581 15478 rpcz_store.cc:276] 0810 08:00:24.215803 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:24.215971 (+ 168us) service_pool.cc:227] Handling call
0810 08:00:25.422875 (+1206904us) ts_tablet_manager.cc:1145] Deleting tablet abdd2b119a4b448dacdd2d87da34832f
0810 08:00:25.422927 (+ 52us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:25.423184 (+ 257us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1205603}
W20260810 08:00:25.431285 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet abdd2b119a4b448dacdd2d87da34832f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abdd2b119a4b448dacdd2d87da34832f already in progress: creating tablet
I20260810 08:00:25.434051 15437 tablet_bootstrap.cc:654] T dc3ea02864d040c5bcde38b1a5609676 P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:25.434702 15478 tablet_replica.cc:330] T 5e0e4f1c76f843ea9000ed25acec73d3 P 3fc5e568f3724f158b616c303efb9eae: stopping tablet replica
I20260810 08:00:25.438251 15478 raft_consensus.cc:2229] T 5e0e4f1c76f843ea9000ed25acec73d3 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:25.462525 15478 raft_consensus.cc:2258] T 5e0e4f1c76f843ea9000ed25acec73d3 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:25.465021 15478 ts_tablet_manager.cc:1905] T 5e0e4f1c76f843ea9000ed25acec73d3 P 3fc5e568f3724f158b616c303efb9eae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:25.475423 14690 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_da20d9f3b8384f3fac7f39669323006b" } new_table_name: "default.Table_dfe235d424c64948a6330b4935504487" modify_external_catalogs: true
I20260810 08:00:25.476518 14689 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_b017ccb238ff4e75afbdf87098ee3bcc" } new_table_name: "default.Table_456df075e34946b686e294549274bbd0" modify_external_catalogs: true
I20260810 08:00:25.494582 14688 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet 114d3eac994f466bb1787e066de05e2c
I20260810 08:00:25.500284 14964 tablet_service.cc:1514] Processing DeleteTablet for tablet 114d3eac994f466bb1787e066de05e2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:25 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:25.502274 14687 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_2373197d513c4475a78e38b1bee008ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:25.504443 14687 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2373197d513c4475a78e38b1bee008ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:25.502297 14691 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_233aa3d7e25f4dae8cbf9e3303c22cf1" } modify_external_catalogs: true
I20260810 08:00:25.507706 14691 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_233aa3d7e25f4dae8cbf9e3303c22cf1" } modify_external_catalogs: true
I20260810 08:00:25.510816 15276 tablet_service.cc:1514] Processing DeleteTablet for tablet 114d3eac994f466bb1787e066de05e2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:25 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:25.511921 15439 tablet_bootstrap.cc:492] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:25.512322 15439 ts_tablet_manager.cc:1397] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.201s user 0.007s sys 0.003s
I20260810 08:00:25.517757 15437 tablet_bootstrap.cc:492] T dc3ea02864d040c5bcde38b1a5609676 P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:25.518190 15437 ts_tablet_manager.cc:1397] T dc3ea02864d040c5bcde38b1a5609676 P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.227s user 0.006s sys 0.010s
I20260810 08:00:25.520542 15437 raft_consensus.cc:348] T dc3ea02864d040c5bcde38b1a5609676 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:25.521207 15437 raft_consensus.cc:374] T dc3ea02864d040c5bcde38b1a5609676 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:25.514647 15439 raft_consensus.cc:348] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:25.522879 15439 raft_consensus.cc:374] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:25.523159 15439 raft_consensus.cc:729] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:25.523913 15439 consensus_queue.cc:260] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:25.529213 14692 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet dc3ea02864d040c5bcde38b1a5609676
I20260810 08:00:25.521448 15437 raft_consensus.cc:729] T dc3ea02864d040c5bcde38b1a5609676 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:25.532141 14692 catalog_manager.cc:6565] ReplaceTablet: tablet dc3ea02864d040c5bcde38b1a5609676 deleted and replaced by tablet f4cafec906564957b01f2227283baccb
I20260810 08:00:25.533397 15276 tablet_service.cc:1514] Processing DeleteTablet for tablet dc3ea02864d040c5bcde38b1a5609676 with delete_type TABLET_DATA_DELETED (replaced by tablet f4cafec906564957b01f2227283baccb) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:25.534756 15437 consensus_queue.cc:260] T dc3ea02864d040c5bcde38b1a5609676 P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:25.556689 15490 ts_tablet_manager.cc:1428] T 9ddae63b29c34e1397c5765e52e1478f P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.221s user 0.005s sys 0.005s
I20260810 08:00:25.562747 15479 ts_tablet_manager.cc:1918] T e0206444ca194f8f98e9d71aa858283d P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:25.563287 15479 log.cc:1196] T e0206444ca194f8f98e9d71aa858283d P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/e0206444ca194f8f98e9d71aa858283d
I20260810 08:00:25.564954 15437 ts_tablet_manager.cc:1428] T dc3ea02864d040c5bcde38b1a5609676 P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.046s user 0.006s sys 0.000s
I20260810 08:00:25.553301 15439 ts_tablet_manager.cc:1428] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.041s user 0.006s sys 0.001s
I20260810 08:00:25.571373 15490 tablet_bootstrap.cc:492] T 472fbc9204754317aab059a4844d7de1 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
W20260810 08:00:25.581105 15234 consensus_peers.cc:487] T 92bbad33a92643e3be3899ace5e29f52 P 3fc5e568f3724f158b616c303efb9eae -> Peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Couldn't send request to peer a91b27762182454baada71e91ceab224. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260810 08:00:25.587152 15439 tablet_bootstrap.cc:492] T 0c152aab7e244afd857534869b13815c P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:25.602609 15437 tablet_bootstrap.cc:492] T b798daf216454a4598437947928803d1 P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:25.626542 15479 ts_tablet_manager.cc:1939] T e0206444ca194f8f98e9d71aa858283d P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:25.636399 15437 tablet_bootstrap.cc:654] T b798daf216454a4598437947928803d1 P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:25.643172 15439 tablet_bootstrap.cc:654] T 0c152aab7e244afd857534869b13815c P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:25.658012 15443 raft_consensus.cc:482] T dc3ea02864d040c5bcde38b1a5609676 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:25.658694 15443 raft_consensus.cc:504] T dc3ea02864d040c5bcde38b1a5609676 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:25.663674 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3ea02864d040c5bcde38b1a5609676" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:25.665740 15443 leader_election.cc:288] T dc3ea02864d040c5bcde38b1a5609676 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:25.668879 15147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3ea02864d040c5bcde38b1a5609676" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:25.657312 15478 ts_tablet_manager.cc:1918] T 5e0e4f1c76f843ea9000ed25acec73d3 P 3fc5e568f3724f158b616c303efb9eae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:25.670095 15147 raft_consensus.cc:2379] T dc3ea02864d040c5bcde38b1a5609676 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a91b27762182454baada71e91ceab224 in current term 1: Already voted for candidate 3fc5e568f3724f158b616c303efb9eae in this term.
I20260810 08:00:25.670595 15478 log.cc:1196] T 5e0e4f1c76f843ea9000ed25acec73d3 P 3fc5e568f3724f158b616c303efb9eae: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal/wals/5e0e4f1c76f843ea9000ed25acec73d3
I20260810 08:00:25.674629 14922 leader_election.cc:302] T dc3ea02864d040c5bcde38b1a5609676 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a91b27762182454baada71e91ceab224; no voters: 3fc5e568f3724f158b616c303efb9eae, 9d7219686f304f62a54acc7497cbbd07
I20260810 08:00:25.676239 15478 ts_tablet_manager.cc:1939] T 5e0e4f1c76f843ea9000ed25acec73d3 P 3fc5e568f3724f158b616c303efb9eae: Deleting consensus metadata
I20260810 08:00:25.677490 15443 raft_consensus.cc:3037] T dc3ea02864d040c5bcde38b1a5609676 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:25.682567 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 25) took 1223 ms. Trace:
I20260810 08:00:25.683176 15126 tablet_service.cc:1514] Processing DeleteTablet for tablet 114d3eac994f466bb1787e066de05e2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:25 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:25.689728 15126 tablet_service.cc:1514] Processing DeleteTablet for tablet dc3ea02864d040c5bcde38b1a5609676 with delete_type TABLET_DATA_DELETED (replaced by tablet f4cafec906564957b01f2227283baccb) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:25.691689 14964 tablet_service.cc:1514] Processing DeleteTablet for tablet dc3ea02864d040c5bcde38b1a5609676 with delete_type TABLET_DATA_DELETED (replaced by tablet f4cafec906564957b01f2227283baccb) from {username='slave'} at 127.0.0.1:34630
W20260810 08:00:25.692701 14678 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 114d3eac994f466bb1787e066de05e2c (table default.Table_6fcf472630534619ae18553e71ebedaf [id=2c633c1a27cf44f3ab04e66e2cba6e13], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 114d3eac994f466bb1787e066de05e2c consensus state has no leader
I20260810 08:00:25.691537 15439 tablet_bootstrap.cc:492] T 0c152aab7e244afd857534869b13815c P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:25.701267 15439 ts_tablet_manager.cc:1397] T 0c152aab7e244afd857534869b13815c P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.114s user 0.010s sys 0.005s
I20260810 08:00:25.698724 15478 rpcz_store.cc:276] 0810 08:00:24.458830 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:24.458977 (+ 147us) service_pool.cc:227] Handling call
0810 08:00:25.434537 (+975560us) ts_tablet_manager.cc:1145] Deleting tablet 5e0e4f1c76f843ea9000ed25acec73d3
0810 08:00:25.434583 (+ 46us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:25.531860 (+ 97277us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:25.656955 (+125095us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:25.682501 (+ 25546us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":974954}
I20260810 08:00:25.708251 15443 raft_consensus.cc:2726] T dc3ea02864d040c5bcde38b1a5609676 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:25.710191 15437 tablet_bootstrap.cc:492] T b798daf216454a4598437947928803d1 P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:25.710927 15437 ts_tablet_manager.cc:1397] T b798daf216454a4598437947928803d1 P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.109s user 0.007s sys 0.008s
I20260810 08:00:25.715358 14678 catalog_manager.cc:4841] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): tablet 5e0e4f1c76f843ea9000ed25acec73d3 (table default.Table_43a063da4167472198ae9533501169eb [id=9cbd826186904ef687327cd9e02fdc1a]) successfully deleted
I20260810 08:00:25.715752 15478 tablet_replica.cc:330] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 3fc5e568f3724f158b616c303efb9eae: stopping tablet replica
I20260810 08:00:25.716822 15478 raft_consensus.cc:2229] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:25.717634 15478 raft_consensus.cc:2258] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:25.704809 15439 raft_consensus.cc:348] T 0c152aab7e244afd857534869b13815c P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:25.723445 15439 raft_consensus.cc:374] T 0c152aab7e244afd857534869b13815c P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:25.723737 15439 raft_consensus.cc:729] T 0c152aab7e244afd857534869b13815c P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:25.629499 15479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43002 (request call id 18) took 1568 ms. Trace:
I20260810 08:00:25.725461 15447 consensus_queue.cc:1035] T dc3ea02864d040c5bcde38b1a5609676 P 3fc5e568f3724f158b616c303efb9eae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 08:00:25.725268 15479 rpcz_store.cc:276] 0810 08:00:24.060899 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:24.061061 (+ 162us) service_pool.cc:227] Handling call
0810 08:00:25.013833 (+952772us) ts_tablet_manager.cc:1145] Deleting tablet e0206444ca194f8f98e9d71aa858283d
0810 08:00:25.013881 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:25.263529 (+249648us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:25.562377 (+298848us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:25.629408 (+ 67031us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":952124}
I20260810 08:00:25.729199 15478 ts_tablet_manager.cc:1905] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 3fc5e568f3724f158b616c303efb9eae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:25.724248 15437 raft_consensus.cc:348] T b798daf216454a4598437947928803d1 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:25.730329 15437 raft_consensus.cc:374] T b798daf216454a4598437947928803d1 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:25.730594 15437 raft_consensus.cc:729] T b798daf216454a4598437947928803d1 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
W20260810 08:00:25.734333 14690 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0fae4809ef5e4a9fb16976674e287f03 has no consensus state
I20260810 08:00:25.736351 15439 consensus_queue.cc:260] T 0c152aab7e244afd857534869b13815c P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:25.743152 15490 tablet_bootstrap.cc:654] T 472fbc9204754317aab059a4844d7de1 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:25.752524 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet e0206444ca194f8f98e9d71aa858283d (table default.Table_0e30c621046242609c9f6fc73b16d0e7 [id=e9608d66a9964bc6aafdf84f574fba02]) successfully deleted
I20260810 08:00:25.757803 14690 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_b7ee9352799b44f4b7bbf48355b3af11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:25.757447 14686 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_1732e711b4dd4f3392ee652299146573"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:25.760141 14690 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7ee9352799b44f4b7bbf48355b3af11 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 08:00:25.761899 14686 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1732e711b4dd4f3392ee652299146573 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:25.764325 15479 tablet_replica.cc:330] T ce99d91e79cf4e06b71110eada58a8e9 P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:25.765403 15479 raft_consensus.cc:2229] T ce99d91e79cf4e06b71110eada58a8e9 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:25.765964 15479 raft_consensus.cc:2258] T ce99d91e79cf4e06b71110eada58a8e9 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:25.742437 15437 consensus_queue.cc:260] T b798daf216454a4598437947928803d1 P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:25.776024 15437 ts_tablet_manager.cc:1428] T b798daf216454a4598437947928803d1 P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.065s user 0.006s sys 0.000s
I20260810 08:00:25.789914 14694 master_service.cc:946] ReplaceTablet: received request to replace tablet 2faab63c5023431c98e7b01ce61e04c2 from {username='slave'} at 127.0.0.1:42300
I20260810 08:00:25.799113 15439 ts_tablet_manager.cc:1428] T 0c152aab7e244afd857534869b13815c P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.097s user 0.006s sys 0.001s
I20260810 08:00:25.799803 15447 raft_consensus.cc:482] T 0c152aab7e244afd857534869b13815c P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:25.800735 15447 raft_consensus.cc:504] T 0c152aab7e244afd857534869b13815c P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:25.771790 15444 raft_consensus.cc:482] T 1461cdeda5664afd9c4c2b45199179b6 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:25.808583 15444 raft_consensus.cc:504] T 1461cdeda5664afd9c4c2b45199179b6 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:25.811522 15444 leader_election.cc:288] T 1461cdeda5664afd9c4c2b45199179b6 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:25.814076 14989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1461cdeda5664afd9c4c2b45199179b6" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
W20260810 08:00:25.815946 15079 leader_election.cc:341] T 1461cdeda5664afd9c4c2b45199179b6 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:25.819200 15439 tablet_bootstrap.cc:492] T b798daf216454a4598437947928803d1 P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
W20260810 08:00:25.827087 14689 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 49c35b7f4a72437fa0675abe8a8f3a71 has no consensus state
I20260810 08:00:25.828073 15437 tablet_bootstrap.cc:492] T 9ddae63b29c34e1397c5765e52e1478f P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:25.832738 15447 leader_election.cc:288] T 0c152aab7e244afd857534869b13815c P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:25.834622 15437 tablet_bootstrap.cc:654] T 9ddae63b29c34e1397c5765e52e1478f P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:25.835512 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1461cdeda5664afd9c4c2b45199179b6" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:25.838757 14989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c152aab7e244afd857534869b13815c" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:25.839597 14989 raft_consensus.cc:2379] T 0c152aab7e244afd857534869b13815c P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fc5e568f3724f158b616c303efb9eae in current term 1: Already voted for candidate 9d7219686f304f62a54acc7497cbbd07 in this term.
I20260810 08:00:25.839754 15479 ts_tablet_manager.cc:1905] T ce99d91e79cf4e06b71110eada58a8e9 P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:25.840065 15437 tablet_bootstrap.cc:492] T 9ddae63b29c34e1397c5765e52e1478f P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:25.840431 15437 ts_tablet_manager.cc:1397] T 9ddae63b29c34e1397c5765e52e1478f P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.063s user 0.006s sys 0.002s
W20260810 08:00:25.841995 15076 leader_election.cc:341] T 1461cdeda5664afd9c4c2b45199179b6 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:25.842589 15076 leader_election.cc:302] T 1461cdeda5664afd9c4c2b45199179b6 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:25.843030 15437 raft_consensus.cc:348] T 9ddae63b29c34e1397c5765e52e1478f P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:25.843719 15437 raft_consensus.cc:374] T 9ddae63b29c34e1397c5765e52e1478f P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:25.843987 15437 raft_consensus.cc:729] T 9ddae63b29c34e1397c5765e52e1478f P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:25.844285 15147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c152aab7e244afd857534869b13815c" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:25.844775 15437 consensus_queue.cc:260] T 9ddae63b29c34e1397c5765e52e1478f P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:25.846427 15234 leader_election.cc:302] T 0c152aab7e244afd857534869b13815c P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae; no voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224
I20260810 08:00:25.846545 15444 raft_consensus.cc:2726] T 1461cdeda5664afd9c4c2b45199179b6 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:25.846837 15437 ts_tablet_manager.cc:1428] T 9ddae63b29c34e1397c5765e52e1478f P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260810 08:00:25.851177 15447 raft_consensus.cc:3037] T 0c152aab7e244afd857534869b13815c P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:25.867030 15439 tablet_bootstrap.cc:654] T b798daf216454a4598437947928803d1 P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:25.876580 14691 catalog_manager.cc:5777] Not sending DeleteTablet requests; no consensus state for tablet f6e641a2a8d74599948338f2b640681d
I20260810 08:00:25.880221 15437 tablet_bootstrap.cc:492] T 1461cdeda5664afd9c4c2b45199179b6 P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:25.887135 15490 tablet_bootstrap.cc:492] T 472fbc9204754317aab059a4844d7de1 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:25.887570 15490 ts_tablet_manager.cc:1397] T 472fbc9204754317aab059a4844d7de1 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.316s user 0.012s sys 0.000s
I20260810 08:00:25.891495 15549 consensus_queue.cc:1035] T dc3ea02864d040c5bcde38b1a5609676 P 3fc5e568f3724f158b616c303efb9eae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 08:00:25.896921 15490 raft_consensus.cc:348] T 472fbc9204754317aab059a4844d7de1 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:25.897643 15490 raft_consensus.cc:374] T 472fbc9204754317aab059a4844d7de1 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:25.897922 15490 raft_consensus.cc:729] T 472fbc9204754317aab059a4844d7de1 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:25.898517 15490 consensus_queue.cc:260] T 472fbc9204754317aab059a4844d7de1 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:25.900476 15490 ts_tablet_manager.cc:1428] T 472fbc9204754317aab059a4844d7de1 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20260810 08:00:25.901296 15490 tablet_bootstrap.cc:492] T a246b8cad99b47a9a856f6aed70bdfc8 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:25.885449 15437 tablet_bootstrap.cc:654] T 1461cdeda5664afd9c4c2b45199179b6 P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:25.910207 15444 raft_consensus.cc:482] T 472fbc9204754317aab059a4844d7de1 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:25.910920 15444 raft_consensus.cc:504] T 472fbc9204754317aab059a4844d7de1 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:25.914529 15444 leader_election.cc:288] T 472fbc9204754317aab059a4844d7de1 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:25.915052 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472fbc9204754317aab059a4844d7de1" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:25.917011 14989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472fbc9204754317aab059a4844d7de1" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
W20260810 08:00:25.918476 15079 leader_election.cc:341] T 472fbc9204754317aab059a4844d7de1 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
W20260810 08:00:25.919559 15076 leader_election.cc:341] T 472fbc9204754317aab059a4844d7de1 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:25.920145 15076 leader_election.cc:302] T 472fbc9204754317aab059a4844d7de1 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:25.921779 15444 raft_consensus.cc:2726] T 472fbc9204754317aab059a4844d7de1 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:25.922925 15437 tablet_bootstrap.cc:492] T 1461cdeda5664afd9c4c2b45199179b6 P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:25.923409 15437 ts_tablet_manager.cc:1397] T 1461cdeda5664afd9c4c2b45199179b6 P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.043s user 0.010s sys 0.000s
I20260810 08:00:25.931967 15490 tablet_bootstrap.cc:654] T a246b8cad99b47a9a856f6aed70bdfc8 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:25.934996 15443 raft_consensus.cc:482] T b798daf216454a4598437947928803d1 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:25.935240 15437 raft_consensus.cc:348] T 1461cdeda5664afd9c4c2b45199179b6 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:25.935510 15443 raft_consensus.cc:504] T b798daf216454a4598437947928803d1 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:25.935952 15437 raft_consensus.cc:374] T 1461cdeda5664afd9c4c2b45199179b6 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:25.936197 15437 raft_consensus.cc:729] T 1461cdeda5664afd9c4c2b45199179b6 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:25.937304 15443 leader_election.cc:288] T b798daf216454a4598437947928803d1 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:25.938647 15437 consensus_queue.cc:260] T 1461cdeda5664afd9c4c2b45199179b6 P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:25.938997 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b798daf216454a4598437947928803d1" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
W20260810 08:00:25.941120 14918 leader_election.cc:341] T b798daf216454a4598437947928803d1 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:25.944401 15146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b798daf216454a4598437947928803d1" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:25.945241 15146 raft_consensus.cc:2454] T b798daf216454a4598437947928803d1 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 0.
I20260810 08:00:25.948586 15437 ts_tablet_manager.cc:1428] T 1461cdeda5664afd9c4c2b45199179b6 P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20260810 08:00:25.948163 14922 leader_election.cc:302] T b798daf216454a4598437947928803d1 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224; no voters: 3fc5e568f3724f158b616c303efb9eae
I20260810 08:00:25.952706 15443 raft_consensus.cc:2781] T b798daf216454a4598437947928803d1 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:25.953164 15443 raft_consensus.cc:482] T b798daf216454a4598437947928803d1 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:25.953480 15443 raft_consensus.cc:3037] T b798daf216454a4598437947928803d1 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:25.954185 15444 raft_consensus.cc:482] T 9ddae63b29c34e1397c5765e52e1478f P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:25.954826 15444 raft_consensus.cc:504] T 9ddae63b29c34e1397c5765e52e1478f P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:25.958240 14989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ddae63b29c34e1397c5765e52e1478f" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:25.958847 14989 raft_consensus.cc:2454] T 9ddae63b29c34e1397c5765e52e1478f P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d7219686f304f62a54acc7497cbbd07 in term 0.
I20260810 08:00:25.960032 15079 leader_election.cc:302] T 9ddae63b29c34e1397c5765e52e1478f P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224; no voters:
I20260810 08:00:25.960840 15541 raft_consensus.cc:2781] T 9ddae63b29c34e1397c5765e52e1478f P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:25.961129 15541 raft_consensus.cc:482] T 9ddae63b29c34e1397c5765e52e1478f P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:25.961421 15541 raft_consensus.cc:3037] T 9ddae63b29c34e1397c5765e52e1478f P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:25.962329 15443 raft_consensus.cc:504] T b798daf216454a4598437947928803d1 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:25.965152 14687 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_bae5256f457944049d86e384fb3759d6" } modify_external_catalogs: true
I20260810 08:00:25.965543 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b798daf216454a4598437947928803d1" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae"
I20260810 08:00:25.965919 14687 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_bae5256f457944049d86e384fb3759d6" } modify_external_catalogs: true
W20260810 08:00:25.966974 14918 leader_election.cc:341] T b798daf216454a4598437947928803d1 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:25.952386 15437 tablet_bootstrap.cc:492] T a246b8cad99b47a9a856f6aed70bdfc8 P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:25.972223 15490 tablet_bootstrap.cc:492] T a246b8cad99b47a9a856f6aed70bdfc8 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:25.972656 15490 ts_tablet_manager.cc:1397] T a246b8cad99b47a9a856f6aed70bdfc8 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.072s user 0.009s sys 0.003s
I20260810 08:00:25.973373 15146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b798daf216454a4598437947928803d1" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07"
I20260810 08:00:25.974063 15146 raft_consensus.cc:3037] T b798daf216454a4598437947928803d1 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:25.974382 15443 leader_election.cc:288] T b798daf216454a4598437947928803d1 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Requested vote from peers 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:25.980262 15439 tablet_bootstrap.cc:492] T b798daf216454a4598437947928803d1 P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:25.980691 15439 ts_tablet_manager.cc:1397] T b798daf216454a4598437947928803d1 P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.162s user 0.009s sys 0.002s
I20260810 08:00:25.980155 15490 raft_consensus.cc:348] T a246b8cad99b47a9a856f6aed70bdfc8 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:25.983557 15490 raft_consensus.cc:374] T a246b8cad99b47a9a856f6aed70bdfc8 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:25.984102 14686 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_1f6f2718c41e46f1b6eb3728857ffbd4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:25.985292 15437 tablet_bootstrap.cc:654] T a246b8cad99b47a9a856f6aed70bdfc8 P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:25.983824 15490 raft_consensus.cc:729] T a246b8cad99b47a9a856f6aed70bdfc8 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:25.986382 15490 consensus_queue.cc:260] T a246b8cad99b47a9a856f6aed70bdfc8 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:25.987217 15478 ts_tablet_manager.cc:1918] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 3fc5e568f3724f158b616c303efb9eae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20260810 08:00:25.987464 14686 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f6f2718c41e46f1b6eb3728857ffbd4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:25.987630 15478 log.cc:1196] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 3fc5e568f3724f158b616c303efb9eae: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal/wals/aeb7c1c4bb1f45f9b46fe5bd78e96a20
I20260810 08:00:25.988451 15478 ts_tablet_manager.cc:1939] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 3fc5e568f3724f158b616c303efb9eae: Deleting consensus metadata
I20260810 08:00:25.950824 15447 raft_consensus.cc:2726] T 0c152aab7e244afd857534869b13815c P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:25.999153 15490 ts_tablet_manager.cc:1428] T a246b8cad99b47a9a856f6aed70bdfc8 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
I20260810 08:00:26.000082 15490 tablet_bootstrap.cc:492] T 2faab63c5023431c98e7b01ce61e04c2 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:26.008718 15444 leader_election.cc:288] T 9ddae63b29c34e1397c5765e52e1478f P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:26.009405 15444 consensus_queue.cc:1035] T 0c152aab7e244afd857534869b13815c P 9d7219686f304f62a54acc7497cbbd07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 08:00:26.010300 14690 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_456df075e34946b686e294549274bbd0" } new_table_name: "default.Table_b78f4082645e4c51ad4d4b487939ffee" modify_external_catalogs: true
I20260810 08:00:26.013873 14691 master_service.cc:946] ReplaceTablet: received request to replace tablet 472fbc9204754317aab059a4844d7de1 from {username='slave'} at 127.0.0.1:42300
I20260810 08:00:26.020649 15490 tablet_bootstrap.cc:654] T 2faab63c5023431c98e7b01ce61e04c2 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:26.024503 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ddae63b29c34e1397c5765e52e1478f" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:26.024923 15146 raft_consensus.cc:2454] T b798daf216454a4598437947928803d1 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 1.
I20260810 08:00:26.026065 14922 leader_election.cc:302] T b798daf216454a4598437947928803d1 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224; no voters: 3fc5e568f3724f158b616c303efb9eae
I20260810 08:00:26.026250 15541 raft_consensus.cc:504] T 9ddae63b29c34e1397c5765e52e1478f P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:26.026943 15443 raft_consensus.cc:2781] T b798daf216454a4598437947928803d1 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 08:00:26.027405 15443 raft_consensus.cc:686] T b798daf216454a4598437947928803d1 P a91b27762182454baada71e91ceab224 [term 1 LEADER]: Becoming Leader. State: Replica: a91b27762182454baada71e91ceab224, State: Running, Role: LEADER
I20260810 08:00:26.028282 15443 consensus_queue.cc:237] T b798daf216454a4598437947928803d1 P a91b27762182454baada71e91ceab224 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:26.033696 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ddae63b29c34e1397c5765e52e1478f" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224"
I20260810 08:00:26.034345 14987 raft_consensus.cc:3037] T 9ddae63b29c34e1397c5765e52e1478f P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:26.034984 15482 ts_tablet_manager.cc:1918] T 92bbad33a92643e3be3899ace5e29f52 P a91b27762182454baada71e91ceab224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:26.035328 15482 log.cc:1196] T 92bbad33a92643e3be3899ace5e29f52 P a91b27762182454baada71e91ceab224: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal/wals/92bbad33a92643e3be3899ace5e29f52
I20260810 08:00:26.035991 15482 ts_tablet_manager.cc:1939] T 92bbad33a92643e3be3899ace5e29f52 P a91b27762182454baada71e91ceab224: Deleting consensus metadata
W20260810 08:00:26.043100 14690 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 49c35b7f4a72437fa0675abe8a8f3a71 has no consensus state
I20260810 08:00:26.045372 14987 raft_consensus.cc:2454] T 9ddae63b29c34e1397c5765e52e1478f P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d7219686f304f62a54acc7497cbbd07 in term 1.
I20260810 08:00:26.050467 14686 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_1f6f2718c41e46f1b6eb3728857ffbd4" } new_table_name: "default.Table_7beb9923b5d5438984557b4df7910362" modify_external_catalogs: true
I20260810 08:00:26.066970 14681 catalog_manager.cc:4841] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): tablet 92bbad33a92643e3be3899ace5e29f52 (table default.Table_7f3a4e19ded64b3783a00b3d4c6839c3 [id=6c6afcf578534997878b98feb9574e2d]) successfully deleted
W20260810 08:00:26.069882 14681 catalog_manager.cc:4821] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet 2faab63c5023431c98e7b01ce61e04c2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2faab63c5023431c98e7b01ce61e04c2 already in progress: creating tablet
I20260810 08:00:26.070993 15482 tablet_replica.cc:330] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P a91b27762182454baada71e91ceab224: stopping tablet replica
I20260810 08:00:26.072031 15482 raft_consensus.cc:2229] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:26.072557 15482 raft_consensus.cc:2258] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:26.074404 15482 ts_tablet_manager.cc:1905] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P a91b27762182454baada71e91ceab224: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 08:00:26.082556 14686 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d0761b090d8147b8a5537b0ed51108a9 has no consensus state
I20260810 08:00:26.083698 15437 tablet_bootstrap.cc:492] T a246b8cad99b47a9a856f6aed70bdfc8 P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:26.084219 15437 ts_tablet_manager.cc:1397] T a246b8cad99b47a9a856f6aed70bdfc8 P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.135s user 0.017s sys 0.011s
I20260810 08:00:26.086118 14686 master_service.cc:946] ReplaceTablet: received request to replace tablet 9ddae63b29c34e1397c5765e52e1478f from {username='slave'} at 127.0.0.1:42300
I20260810 08:00:26.086545 15437 raft_consensus.cc:348] T a246b8cad99b47a9a856f6aed70bdfc8 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:26.087212 15437 raft_consensus.cc:374] T a246b8cad99b47a9a856f6aed70bdfc8 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:26.087467 15437 raft_consensus.cc:729] T a246b8cad99b47a9a856f6aed70bdfc8 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:26.088133 15437 consensus_queue.cc:260] T a246b8cad99b47a9a856f6aed70bdfc8 P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:26.090473 15437 ts_tablet_manager.cc:1428] T a246b8cad99b47a9a856f6aed70bdfc8 P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 08:00:26.093139 15437 tablet_bootstrap.cc:492] T 472fbc9204754317aab059a4844d7de1 P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:26.095423 15439 raft_consensus.cc:348] T b798daf216454a4598437947928803d1 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:26.096199 15439 raft_consensus.cc:374] T b798daf216454a4598437947928803d1 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:26.096531 15439 raft_consensus.cc:729] T b798daf216454a4598437947928803d1 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:26.100291 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 28) took 1417 ms. Trace:
I20260810 08:00:26.100739 15478 rpcz_store.cc:276] 0810 08:00:24.682546 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:24.685712 (+ 3166us) service_pool.cc:227] Handling call
0810 08:00:25.715458 (+1029746us) ts_tablet_manager.cc:1145] Deleting tablet aeb7c1c4bb1f45f9b46fe5bd78e96a20
0810 08:00:25.715512 (+ 54us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:25.832496 (+116984us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:25.986986 (+154490us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:26.100227 (+113241us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":109910,"tablet-delete.queue_time_us":1029208}
I20260810 08:00:26.102142 14678 catalog_manager.cc:4841] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): tablet aeb7c1c4bb1f45f9b46fe5bd78e96a20 (table default.Table_9b0d38b98f3841828a00a23dda57ae6e [id=40cf37d4ad57413fa64ee743a5c160c3]) successfully deleted
I20260810 08:00:26.103233 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 29) took 1199 ms. Trace:
I20260810 08:00:26.103641 15478 rpcz_store.cc:276] 0810 08:00:24.903490 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:24.903638 (+ 148us) service_pool.cc:227] Handling call
0810 08:00:26.102883 (+1199245us) ts_tablet_manager.cc:1145] Deleting tablet a246b8cad99b47a9a856f6aed70bdfc8
0810 08:00:26.102930 (+ 47us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:26.103180 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1197725}
W20260810 08:00:26.105227 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet a246b8cad99b47a9a856f6aed70bdfc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a246b8cad99b47a9a856f6aed70bdfc8 already in progress: creating tablet
I20260810 08:00:26.106123 15478 tablet_replica.cc:330] T 92bbad33a92643e3be3899ace5e29f52 P 3fc5e568f3724f158b616c303efb9eae: stopping tablet replica
I20260810 08:00:26.106918 15478 raft_consensus.cc:2229] T 92bbad33a92643e3be3899ace5e29f52 P 3fc5e568f3724f158b616c303efb9eae [term 1 LEADER]: Raft consensus shutting down.
I20260810 08:00:26.107887 15478 raft_consensus.cc:2258] T 92bbad33a92643e3be3899ace5e29f52 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:26.097256 15439 consensus_queue.cc:260] T b798daf216454a4598437947928803d1 P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:26.111371 15478 ts_tablet_manager.cc:1905] T 92bbad33a92643e3be3899ace5e29f52 P 3fc5e568f3724f158b616c303efb9eae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:26.129055 15439 ts_tablet_manager.cc:1428] T b798daf216454a4598437947928803d1 P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.148s user 0.008s sys 0.000s
I20260810 08:00:26.131834 14695 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_409addd050ee45b981bddfacfab24c03"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:26.137879 15439 tablet_bootstrap.cc:492] T abdd2b119a4b448dacdd2d87da34832f P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:26.143225 15437 tablet_bootstrap.cc:654] T 472fbc9204754317aab059a4844d7de1 P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:26.145306 15482 ts_tablet_manager.cc:1918] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P a91b27762182454baada71e91ceab224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:26.145689 15482 log.cc:1196] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P a91b27762182454baada71e91ceab224: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal/wals/aeb7c1c4bb1f45f9b46fe5bd78e96a20
I20260810 08:00:26.146308 15482 ts_tablet_manager.cc:1939] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P a91b27762182454baada71e91ceab224: Deleting consensus metadata
I20260810 08:00:26.154665 14692 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_f0a0a164132a4d46891036b0fee9839a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:26.154908 14690 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_64e1c230e9624fc9ba6c328346150ffe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:26.156653 14692 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f0a0a164132a4d46891036b0fee9839a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 08:00:26.156705 14690 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_64e1c230e9624fc9ba6c328346150ffe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:26.165205 15439 tablet_bootstrap.cc:654] T abdd2b119a4b448dacdd2d87da34832f P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:26.175048 15482 tablet_replica.cc:330] T dc3ea02864d040c5bcde38b1a5609676 P a91b27762182454baada71e91ceab224: stopping tablet replica
I20260810 08:00:26.175345 15276 tablet_service.cc:1467] Processing CreateTablet for tablet 0fae4809ef5e4a9fb16976674e287f03 (DEFAULT_TABLE table=default.Table_dfe235d424c64948a6330b4935504487 [id=41bcbdf2ef674d8b8db9284b4b0dcdb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:26.176558 15276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fae4809ef5e4a9fb16976674e287f03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:26.178961 15275 tablet_service.cc:1467] Processing CreateTablet for tablet e46648198f5343f7a65079887e5df159 (DEFAULT_TABLE table=default.Table_dbbc4dceb17d4aecb41b3ec5fc8c21ed [id=a705653aa3484a69b369e7e39273d28e]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 08:00:26.182224 14695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_409addd050ee45b981bddfacfab24c03 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:26.182619 15275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e46648198f5343f7a65079887e5df159. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:26.181728 14964 tablet_service.cc:1467] Processing CreateTablet for tablet 0fae4809ef5e4a9fb16976674e287f03 (DEFAULT_TABLE table=default.Table_dfe235d424c64948a6330b4935504487 [id=41bcbdf2ef674d8b8db9284b4b0dcdb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:26.188464 15482 raft_consensus.cc:2229] T dc3ea02864d040c5bcde38b1a5609676 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:26.188659 14964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fae4809ef5e4a9fb16976674e287f03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:26.188963 15482 raft_consensus.cc:2258] T dc3ea02864d040c5bcde38b1a5609676 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:26.181457 14681 catalog_manager.cc:4841] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): tablet aeb7c1c4bb1f45f9b46fe5bd78e96a20 (table default.Table_9b0d38b98f3841828a00a23dda57ae6e [id=40cf37d4ad57413fa64ee743a5c160c3]) successfully deleted
W20260810 08:00:26.190412 14681 catalog_manager.cc:4821] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet 114d3eac994f466bb1787e066de05e2c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 114d3eac994f466bb1787e066de05e2c already in progress: creating tablet
I20260810 08:00:26.191454 15482 ts_tablet_manager.cc:1905] T dc3ea02864d040c5bcde38b1a5609676 P a91b27762182454baada71e91ceab224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:26.185998 14967 tablet_service.cc:1467] Processing CreateTablet for tablet e46648198f5343f7a65079887e5df159 (DEFAULT_TABLE table=default.Table_dbbc4dceb17d4aecb41b3ec5fc8c21ed [id=a705653aa3484a69b369e7e39273d28e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:26.193889 14967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e46648198f5343f7a65079887e5df159. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:26.196666 14686 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet 9ddae63b29c34e1397c5765e52e1478f
I20260810 08:00:26.199255 15274 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ddae63b29c34e1397c5765e52e1478f with delete_type TABLET_DATA_DELETED (replaced by tablet 561cbc6a4eb14d3584f88ea26a0fea7a) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:26.199972 14686 catalog_manager.cc:6565] ReplaceTablet: tablet 9ddae63b29c34e1397c5765e52e1478f deleted and replaced by tablet 561cbc6a4eb14d3584f88ea26a0fea7a
I20260810 08:00:26.202283 14691 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet 472fbc9204754317aab059a4844d7de1
I20260810 08:00:26.202901 14687 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet 37d8a37a16f34a0285c43c0d71b5f072
I20260810 08:00:26.203928 14691 catalog_manager.cc:6565] ReplaceTablet: tablet 472fbc9204754317aab059a4844d7de1 deleted and replaced by tablet b5a0e0af742340b2945be2102768b307
I20260810 08:00:26.204965 15279 tablet_service.cc:1514] Processing DeleteTablet for tablet 37d8a37a16f34a0285c43c0d71b5f072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:25 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:26.205667 14966 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ddae63b29c34e1397c5765e52e1478f with delete_type TABLET_DATA_DELETED (replaced by tablet 561cbc6a4eb14d3584f88ea26a0fea7a) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:26.206418 15274 tablet_service.cc:1514] Processing DeleteTablet for tablet 472fbc9204754317aab059a4844d7de1 with delete_type TABLET_DATA_DELETED (replaced by tablet b5a0e0af742340b2945be2102768b307) from {username='slave'} at 127.0.0.1:40692
W20260810 08:00:26.042766 15076 leader_election.cc:341] T 9ddae63b29c34e1397c5765e52e1478f P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:26.110880 15573 raft_consensus.cc:482] T a246b8cad99b47a9a856f6aed70bdfc8 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:26.208969 14966 tablet_service.cc:1514] Processing DeleteTablet for tablet 472fbc9204754317aab059a4844d7de1 with delete_type TABLET_DATA_DELETED (replaced by tablet b5a0e0af742340b2945be2102768b307) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:26.208195 14688 catalog_manager.cc:5495] T b798daf216454a4598437947928803d1 P a91b27762182454baada71e91ceab224 reported cstate change: term changed from 0 to 1, leader changed from <none> to a91b27762182454baada71e91ceab224 (127.14.57.193). New cstate: current_term: 1 leader_uuid: "a91b27762182454baada71e91ceab224" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } health_report { overall_health: UNKNOWN } } }
I20260810 08:00:26.208730 15573 raft_consensus.cc:504] T a246b8cad99b47a9a856f6aed70bdfc8 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:26.211578 15573 leader_election.cc:288] T a246b8cad99b47a9a856f6aed70bdfc8 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:26.212706 14966 tablet_service.cc:1514] Processing DeleteTablet for tablet 37d8a37a16f34a0285c43c0d71b5f072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:25 UTC) from {username='slave'} at 127.0.0.1:34630
W20260810 08:00:26.213459 15076 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.195:37729, user_credentials={real_user=slave}} blocked reactor thread for 170751us
I20260810 08:00:26.054638 15079 leader_election.cc:302] T 9ddae63b29c34e1397c5765e52e1478f P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224; no voters:
I20260810 08:00:26.215229 15573 raft_consensus.cc:2781] T 9ddae63b29c34e1397c5765e52e1478f P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 08:00:26.215668 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a246b8cad99b47a9a856f6aed70bdfc8" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:26.215793 15573 raft_consensus.cc:686] T 9ddae63b29c34e1397c5765e52e1478f P 9d7219686f304f62a54acc7497cbbd07 [term 1 LEADER]: Becoming Leader. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Running, Role: LEADER
I20260810 08:00:26.216949 15573 consensus_queue.cc:237] T 9ddae63b29c34e1397c5765e52e1478f P 9d7219686f304f62a54acc7497cbbd07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:26.219450 15478 ts_tablet_manager.cc:1918] T 92bbad33a92643e3be3899ace5e29f52 P 3fc5e568f3724f158b616c303efb9eae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 08:00:26.219810 15478 log.cc:1196] T 92bbad33a92643e3be3899ace5e29f52 P 3fc5e568f3724f158b616c303efb9eae: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal/wals/92bbad33a92643e3be3899ace5e29f52
I20260810 08:00:26.220527 15478 ts_tablet_manager.cc:1939] T 92bbad33a92643e3be3899ace5e29f52 P 3fc5e568f3724f158b616c303efb9eae: Deleting consensus metadata
I20260810 08:00:26.221735 15541 leader_election.cc:288] T 9ddae63b29c34e1397c5765e52e1478f P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 election: Requested vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:26.222819 14689 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_6e428ff2c4c7459b84d2882a089b490d" } modify_external_catalogs: true
I20260810 08:00:26.223534 14689 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_6e428ff2c4c7459b84d2882a089b490d" } modify_external_catalogs: true
I20260810 08:00:26.226728 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ddae63b29c34e1397c5765e52e1478f" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae"
W20260810 08:00:26.228217 15076 leader_election.cc:341] T a246b8cad99b47a9a856f6aed70bdfc8 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
W20260810 08:00:26.228981 15076 leader_election.cc:341] T 9ddae63b29c34e1397c5765e52e1478f P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
W20260810 08:00:26.238547 15079 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.193:43601, user_credentials={real_user=slave}} blocked reactor thread for 185279us
I20260810 08:00:26.241920 15126 tablet_service.cc:1467] Processing CreateTablet for tablet 0fae4809ef5e4a9fb16976674e287f03 (DEFAULT_TABLE table=default.Table_dfe235d424c64948a6330b4935504487 [id=41bcbdf2ef674d8b8db9284b4b0dcdb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:26.243389 15126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fae4809ef5e4a9fb16976674e287f03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:26.250455 14691 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_2373197d513c4475a78e38b1bee008ce" } new_table_name: "default.Table_d5c08115c8b24ad48a9686e48a45ab82" modify_external_catalogs: true
I20260810 08:00:26.255570 14690 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_870c8e79c4574433bf1b38ff75564dc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:26.257318 14690 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_870c8e79c4574433bf1b38ff75564dc9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:26.266680 15439 tablet_bootstrap.cc:492] T abdd2b119a4b448dacdd2d87da34832f P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:26.267117 15439 ts_tablet_manager.cc:1397] T abdd2b119a4b448dacdd2d87da34832f P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.129s user 0.008s sys 0.000s
I20260810 08:00:26.267320 15123 tablet_service.cc:1467] Processing CreateTablet for tablet e46648198f5343f7a65079887e5df159 (DEFAULT_TABLE table=default.Table_dbbc4dceb17d4aecb41b3ec5fc8c21ed [id=a705653aa3484a69b369e7e39273d28e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:26.268603 15123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e46648198f5343f7a65079887e5df159. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:26.271932 15122 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ddae63b29c34e1397c5765e52e1478f with delete_type TABLET_DATA_DELETED (replaced by tablet 561cbc6a4eb14d3584f88ea26a0fea7a) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:26.278816 15274 tablet_service.cc:1467] Processing CreateTablet for tablet 728f8f80b3bb4b10871d8b400f25377b (DEFAULT_TABLE table=default.Table_4c0b6e5e875f43e088f4f0215cf39284 [id=b89eea0100b945c1851d323ce8c97591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:26.281994 15122 tablet_service.cc:1514] Processing DeleteTablet for tablet 472fbc9204754317aab059a4844d7de1 with delete_type TABLET_DATA_DELETED (replaced by tablet b5a0e0af742340b2945be2102768b307) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:26.286118 14692 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_9b2d00d70a97483bb26390a4d1e2bb2f" } new_table_name: "default.Table_5a6ab9b9acc9408c8811f77acaca9a0c" modify_external_catalogs: true
I20260810 08:00:26.286825 15122 tablet_service.cc:1514] Processing DeleteTablet for tablet 37d8a37a16f34a0285c43c0d71b5f072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:25 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:26.294701 15122 tablet_service.cc:1467] Processing CreateTablet for tablet 728f8f80b3bb4b10871d8b400f25377b (DEFAULT_TABLE table=default.Table_4c0b6e5e875f43e088f4f0215cf39284 [id=b89eea0100b945c1851d323ce8c97591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:26.295823 15122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 728f8f80b3bb4b10871d8b400f25377b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:26.298445 15490 tablet_bootstrap.cc:492] T 2faab63c5023431c98e7b01ce61e04c2 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:26.298920 15490 ts_tablet_manager.cc:1397] T 2faab63c5023431c98e7b01ce61e04c2 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.299s user 0.019s sys 0.035s
I20260810 08:00:26.306321 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 32) took 1228 ms. Trace:
I20260810 08:00:26.306777 15478 rpcz_store.cc:276] 0810 08:00:25.078192 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:25.078364 (+ 172us) service_pool.cc:227] Handling call
0810 08:00:26.105971 (+1027607us) ts_tablet_manager.cc:1145] Deleting tablet 92bbad33a92643e3be3899ace5e29f52
0810 08:00:26.106013 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:26.119467 (+ 13454us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:26.219242 (+ 99775us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:26.306248 (+ 87006us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1027000}
I20260810 08:00:26.307926 14678 catalog_manager.cc:4841] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): tablet 92bbad33a92643e3be3899ace5e29f52 (table default.Table_7f3a4e19ded64b3783a00b3d4c6839c3 [id=6c6afcf578534997878b98feb9574e2d]) successfully deleted
I20260810 08:00:26.309170 15490 raft_consensus.cc:348] T 2faab63c5023431c98e7b01ce61e04c2 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:26.311060 15439 raft_consensus.cc:348] T abdd2b119a4b448dacdd2d87da34832f P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:26.311754 15439 raft_consensus.cc:374] T abdd2b119a4b448dacdd2d87da34832f P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:26.312013 15439 raft_consensus.cc:729] T abdd2b119a4b448dacdd2d87da34832f P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:26.314270 15274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 728f8f80b3bb4b10871d8b400f25377b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:26.324009 15275 tablet_service.cc:1467] Processing CreateTablet for tablet 37d8a37a16f34a0285c43c0d71b5f072 (DEFAULT_TABLE table=default.Table_bae5256f457944049d86e384fb3759d6 [id=dad0c6e295df4fdd9b2c27d2b19fbc5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:26.325155 15275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37d8a37a16f34a0285c43c0d71b5f072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:26.242012 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a246b8cad99b47a9a856f6aed70bdfc8" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:26.325930 14987 raft_consensus.cc:2454] T a246b8cad99b47a9a856f6aed70bdfc8 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d7219686f304f62a54acc7497cbbd07 in term 0.
I20260810 08:00:26.335795 14963 tablet_service.cc:1467] Processing CreateTablet for tablet 49c35b7f4a72437fa0675abe8a8f3a71 (DEFAULT_TABLE table=default.Table_b78f4082645e4c51ad4d4b487939ffee [id=6cf63d3e931a4f6487074494254805bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:26.336462 14966 tablet_service.cc:1467] Processing CreateTablet for tablet 728f8f80b3bb4b10871d8b400f25377b (DEFAULT_TABLE table=default.Table_4c0b6e5e875f43e088f4f0215cf39284 [id=b89eea0100b945c1851d323ce8c97591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:26.336948 14963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49c35b7f4a72437fa0675abe8a8f3a71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:26.337524 14966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 728f8f80b3bb4b10871d8b400f25377b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:26.339004 14965 tablet_service.cc:1514] Processing DeleteTablet for tablet a246b8cad99b47a9a856f6aed70bdfc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:24 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:26.340245 14962 tablet_service.cc:1467] Processing CreateTablet for tablet 37d8a37a16f34a0285c43c0d71b5f072 (DEFAULT_TABLE table=default.Table_bae5256f457944049d86e384fb3759d6 [id=dad0c6e295df4fdd9b2c27d2b19fbc5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:26.342033 15279 tablet_service.cc:1467] Processing CreateTablet for tablet 49c35b7f4a72437fa0675abe8a8f3a71 (DEFAULT_TABLE table=default.Table_b78f4082645e4c51ad4d4b487939ffee [id=6cf63d3e931a4f6487074494254805bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:26.342552 14962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37d8a37a16f34a0285c43c0d71b5f072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:26.343214 15279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49c35b7f4a72437fa0675abe8a8f3a71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:26.347337 15490 raft_consensus.cc:374] T 2faab63c5023431c98e7b01ce61e04c2 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:26.347745 15490 raft_consensus.cc:729] T 2faab63c5023431c98e7b01ce61e04c2 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:26.348510 15490 consensus_queue.cc:260] T 2faab63c5023431c98e7b01ce61e04c2 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:26.343854 15079 leader_election.cc:302] T a246b8cad99b47a9a856f6aed70bdfc8 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224; no voters: 3fc5e568f3724f158b616c303efb9eae
W20260810 08:00:26.359325 14691 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2d295305b5a843eaa84c8ad666355b84 has no consensus state
I20260810 08:00:26.367518 15490 ts_tablet_manager.cc:1428] T 2faab63c5023431c98e7b01ce61e04c2 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.068s user 0.003s sys 0.002s
I20260810 08:00:26.368862 15541 raft_consensus.cc:2781] T a246b8cad99b47a9a856f6aed70bdfc8 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:26.369206 15541 raft_consensus.cc:482] T a246b8cad99b47a9a856f6aed70bdfc8 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:26.369557 15541 raft_consensus.cc:3037] T a246b8cad99b47a9a856f6aed70bdfc8 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:26.379343 15126 tablet_service.cc:1467] Processing CreateTablet for tablet 37d8a37a16f34a0285c43c0d71b5f072 (DEFAULT_TABLE table=default.Table_bae5256f457944049d86e384fb3759d6 [id=dad0c6e295df4fdd9b2c27d2b19fbc5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:26.380509 15126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37d8a37a16f34a0285c43c0d71b5f072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:26.380839 15479 ts_tablet_manager.cc:1918] T ce99d91e79cf4e06b71110eada58a8e9 P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:26.381189 15479 log.cc:1196] T ce99d91e79cf4e06b71110eada58a8e9 P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/ce99d91e79cf4e06b71110eada58a8e9
I20260810 08:00:26.381886 15479 ts_tablet_manager.cc:1939] T ce99d91e79cf4e06b71110eada58a8e9 P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:26.382040 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 33) took 1286 ms. Trace:
I20260810 08:00:26.382383 15478 rpcz_store.cc:276] 0810 08:00:25.096111 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:25.096270 (+ 159us) service_pool.cc:227] Handling call
0810 08:00:26.309414 (+1213144us) ts_tablet_manager.cc:1145] Deleting tablet 2faab63c5023431c98e7b01ce61e04c2
0810 08:00:26.381681 (+ 72267us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:26.381976 (+ 295us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1212503}
I20260810 08:00:26.382629 15541 raft_consensus.cc:504] T a246b8cad99b47a9a856f6aed70bdfc8 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
W20260810 08:00:26.383554 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet 2faab63c5023431c98e7b01ce61e04c2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2faab63c5023431c98e7b01ce61e04c2 already in progress: creating tablet
I20260810 08:00:26.385592 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a246b8cad99b47a9a856f6aed70bdfc8" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae"
W20260810 08:00:26.387172 15076 leader_election.cc:341] T a246b8cad99b47a9a856f6aed70bdfc8 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:26.388629 15541 leader_election.cc:288] T a246b8cad99b47a9a856f6aed70bdfc8 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 election: Requested vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:26.392800 14965 tablet_service.cc:1467] Processing CreateTablet for tablet fbf62f4eab8b49f282df3ab4d95d886f (DEFAULT_TABLE table=default.Table_7f3a4e19ded64b3783a00b3d4c6839c3 [id=6c6afcf578534997878b98feb9574e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:26.401708 15122 tablet_service.cc:1467] Processing CreateTablet for tablet 49c35b7f4a72437fa0675abe8a8f3a71 (DEFAULT_TABLE table=default.Table_b78f4082645e4c51ad4d4b487939ffee [id=6cf63d3e931a4f6487074494254805bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:26.403264 14965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbf62f4eab8b49f282df3ab4d95d886f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:26.403231 15122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49c35b7f4a72437fa0675abe8a8f3a71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:26.405766 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a246b8cad99b47a9a856f6aed70bdfc8" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224"
I20260810 08:00:26.406401 14987 raft_consensus.cc:3037] T a246b8cad99b47a9a856f6aed70bdfc8 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:26.418167 15278 tablet_service.cc:1467] Processing CreateTablet for tablet fbf62f4eab8b49f282df3ab4d95d886f (DEFAULT_TABLE table=default.Table_7f3a4e19ded64b3783a00b3d4c6839c3 [id=6c6afcf578534997878b98feb9574e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:26.419066 15443 raft_consensus.cc:482] T 1461cdeda5664afd9c4c2b45199179b6 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:26.419662 15443 raft_consensus.cc:504] T 1461cdeda5664afd9c4c2b45199179b6 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:26.421469 14691 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_6d487e53be1d46deabb8baa50b2f220e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:26.422592 15443 leader_election.cc:288] T 1461cdeda5664afd9c4c2b45199179b6 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:26.424861 14694 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_4008644ee3004c9382c61670dd08b89e" } modify_external_catalogs: true
I20260810 08:00:26.426715 14694 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_4008644ee3004c9382c61670dd08b89e" } modify_external_catalogs: true
I20260810 08:00:26.426383 15573 consensus_queue.cc:1035] T 0c152aab7e244afd857534869b13815c P 9d7219686f304f62a54acc7497cbbd07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 08:00:26.428918 15490 tablet_bootstrap.cc:492] T 114d3eac994f466bb1787e066de05e2c P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
W20260810 08:00:26.431674 14691 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d487e53be1d46deabb8baa50b2f220e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:26.434248 14987 raft_consensus.cc:2454] T a246b8cad99b47a9a856f6aed70bdfc8 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d7219686f304f62a54acc7497cbbd07 in term 1.
I20260810 08:00:26.436713 15479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43002 (request call id 20) took 2293 ms. Trace:
I20260810 08:00:26.437291 15479 rpcz_store.cc:276] 0810 08:00:24.143297 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:24.143602 (+ 305us) service_pool.cc:227] Handling call
0810 08:00:25.727276 (+1583674us) ts_tablet_manager.cc:1145] Deleting tablet ce99d91e79cf4e06b71110eada58a8e9
0810 08:00:25.727344 (+ 68us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:26.229301 (+501957us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:26.380595 (+151294us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:26.436628 (+ 56033us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":51543,"spinlock_wait_cycles":6400,"tablet-delete.queue_time_us":1583061}
I20260810 08:00:26.440918 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1461cdeda5664afd9c4c2b45199179b6" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:26.441109 15079 leader_election.cc:302] T a246b8cad99b47a9a856f6aed70bdfc8 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224; no voters: 3fc5e568f3724f158b616c303efb9eae
I20260810 08:00:26.441469 15278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbf62f4eab8b49f282df3ab4d95d886f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:26.442022 14688 master_service.cc:946] ReplaceTablet: received request to replace tablet 0c152aab7e244afd857534869b13815c from {username='slave'} at 127.0.0.1:42300
I20260810 08:00:26.442749 15478 tablet_replica.cc:330] T a074a741235e4c4b8db1fdb45fb971a8 P 3fc5e568f3724f158b616c303efb9eae: stopping tablet replica
I20260810 08:00:26.443341 15482 ts_tablet_manager.cc:1918] T dc3ea02864d040c5bcde38b1a5609676 P a91b27762182454baada71e91ceab224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 08:00:26.443724 15482 log.cc:1196] T dc3ea02864d040c5bcde38b1a5609676 P a91b27762182454baada71e91ceab224: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal/wals/dc3ea02864d040c5bcde38b1a5609676
I20260810 08:00:26.443642 15126 tablet_service.cc:1467] Processing CreateTablet for tablet fbf62f4eab8b49f282df3ab4d95d886f (DEFAULT_TABLE table=default.Table_7f3a4e19ded64b3783a00b3d4c6839c3 [id=6c6afcf578534997878b98feb9574e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:26.444489 15482 ts_tablet_manager.cc:1939] T dc3ea02864d040c5bcde38b1a5609676 P a91b27762182454baada71e91ceab224: Deleting consensus metadata
I20260810 08:00:26.444736 15126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbf62f4eab8b49f282df3ab4d95d886f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:26.446014 15147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1461cdeda5664afd9c4c2b45199179b6" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:26.446652 15147 raft_consensus.cc:2454] T 1461cdeda5664afd9c4c2b45199179b6 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 0.
I20260810 08:00:26.445958 15478 raft_consensus.cc:2229] T a074a741235e4c4b8db1fdb45fb971a8 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:26.448648 15478 raft_consensus.cc:2258] T a074a741235e4c4b8db1fdb45fb971a8 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:26.346589 15439 consensus_queue.cc:260] T abdd2b119a4b448dacdd2d87da34832f P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:26.452196 15478 ts_tablet_manager.cc:1905] T a074a741235e4c4b8db1fdb45fb971a8 P 3fc5e568f3724f158b616c303efb9eae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:26.469897 15573 raft_consensus.cc:2781] T a246b8cad99b47a9a856f6aed70bdfc8 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 08:00:26.470762 15573 raft_consensus.cc:686] T a246b8cad99b47a9a856f6aed70bdfc8 P 9d7219686f304f62a54acc7497cbbd07 [term 1 LEADER]: Becoming Leader. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Running, Role: LEADER
I20260810 08:00:26.472043 15573 consensus_queue.cc:237] T a246b8cad99b47a9a856f6aed70bdfc8 P 9d7219686f304f62a54acc7497cbbd07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:26.479657 14681 catalog_manager.cc:4841] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): tablet dc3ea02864d040c5bcde38b1a5609676 (table default.Table_4008644ee3004c9382c61670dd08b89e [id=becf1f7969f041d2a6c986f8918187a8]) successfully deleted
I20260810 08:00:26.485440 14693 master_service.cc:946] ReplaceTablet: received request to replace tablet 35b6da60a5544d7e9b657a5f1e6f90ff from {username='slave'} at 127.0.0.1:42300
W20260810 08:00:26.486598 14918 leader_election.cc:341] T 1461cdeda5664afd9c4c2b45199179b6 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:26.490321 15479 tablet_replica.cc:330] T abdd2b119a4b448dacdd2d87da34832f P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:26.491232 15479 raft_consensus.cc:2229] T abdd2b119a4b448dacdd2d87da34832f P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:26.491923 15479 raft_consensus.cc:2258] T abdd2b119a4b448dacdd2d87da34832f P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:26.499290 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet ce99d91e79cf4e06b71110eada58a8e9 (table default.Table_82804e0d8405498b86dd27530bf9384f [id=d1c97b8884c844f8b808baaaf7038af8]) successfully deleted
I20260810 08:00:26.501986 15278 tablet_service.cc:1514] Processing DeleteTablet for tablet abdd2b119a4b448dacdd2d87da34832f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:24 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:26.503996 15478 ts_tablet_manager.cc:1918] T a074a741235e4c4b8db1fdb45fb971a8 P 3fc5e568f3724f158b616c303efb9eae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:26.504400 15478 log.cc:1196] T a074a741235e4c4b8db1fdb45fb971a8 P 3fc5e568f3724f158b616c303efb9eae: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal/wals/a074a741235e4c4b8db1fdb45fb971a8
I20260810 08:00:26.505065 15478 ts_tablet_manager.cc:1939] T a074a741235e4c4b8db1fdb45fb971a8 P 3fc5e568f3724f158b616c303efb9eae: Deleting consensus metadata
I20260810 08:00:26.506973 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 37) took 1205 ms. Trace:
I20260810 08:00:26.507566 15490 tablet_bootstrap.cc:654] T 114d3eac994f466bb1787e066de05e2c P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:26.507266 15478 rpcz_store.cc:276] 0810 08:00:25.301669 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:25.313707 (+ 12038us) service_pool.cc:227] Handling call
0810 08:00:26.411978 (+1098271us) ts_tablet_manager.cc:1145] Deleting tablet a074a741235e4c4b8db1fdb45fb971a8
0810 08:00:26.412031 (+ 53us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:26.473390 (+ 61359us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:26.503741 (+ 30351us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:26.506918 (+ 3177us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1097570}
I20260810 08:00:26.512743 15439 ts_tablet_manager.cc:1428] T abdd2b119a4b448dacdd2d87da34832f P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.245s user 0.009s sys 0.000s
I20260810 08:00:26.516165 14686 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_93c814deac714a3288f3980340acbc65"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:26.518672 14686 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93c814deac714a3288f3980340acbc65 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:26.522491 14695 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_64bd369e459e44d9ac85bf7defb3c86c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:26.524101 14695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_64bd369e459e44d9ac85bf7defb3c86c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:26.525609 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 38) took 1015 ms. Trace:
I20260810 08:00:26.490856 14922 leader_election.cc:302] T 1461cdeda5664afd9c4c2b45199179b6 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224; no voters: 3fc5e568f3724f158b616c303efb9eae
I20260810 08:00:26.524912 14678 catalog_manager.cc:4841] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): tablet a074a741235e4c4b8db1fdb45fb971a8 (table default.Table_dcbe00e111e246a3975b5a4bd8cce1a0 [id=72cc95ab8de44814aaeece045b4d8d77]) successfully deleted
I20260810 08:00:26.525959 15447 raft_consensus.cc:482] T b798daf216454a4598437947928803d1 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:26.531391 15478 rpcz_store.cc:276] 0810 08:00:25.510544 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:25.510666 (+ 122us) service_pool.cc:227] Handling call
0810 08:00:26.525256 (+1014590us) ts_tablet_manager.cc:1145] Deleting tablet 114d3eac994f466bb1787e066de05e2c
0810 08:00:26.525297 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:26.525547 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1014014}
I20260810 08:00:26.531544 15447 raft_consensus.cc:504] T b798daf216454a4598437947928803d1 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
W20260810 08:00:26.532527 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet 114d3eac994f466bb1787e066de05e2c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 114d3eac994f466bb1787e066de05e2c already in progress: creating tablet
I20260810 08:00:26.532747 15478 tablet_replica.cc:330] T dc3ea02864d040c5bcde38b1a5609676 P 3fc5e568f3724f158b616c303efb9eae: stopping tablet replica
I20260810 08:00:26.533401 15478 raft_consensus.cc:2229] T dc3ea02864d040c5bcde38b1a5609676 P 3fc5e568f3724f158b616c303efb9eae [term 1 LEADER]: Raft consensus shutting down.
I20260810 08:00:26.533475 15447 leader_election.cc:288] T b798daf216454a4598437947928803d1 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:26.535327 15147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b798daf216454a4598437947928803d1" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:26.535794 15479 ts_tablet_manager.cc:1905] T abdd2b119a4b448dacdd2d87da34832f P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:26.536187 15147 raft_consensus.cc:2379] T b798daf216454a4598437947928803d1 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fc5e568f3724f158b616c303efb9eae in current term 1: Already voted for candidate a91b27762182454baada71e91ceab224 in this term.
I20260810 08:00:26.534252 15478 raft_consensus.cc:2258] T dc3ea02864d040c5bcde38b1a5609676 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 08:00:26.545892 14922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.194:36265, user_credentials={real_user=slave}} blocked reactor thread for 55101.6us
I20260810 08:00:26.549348 14989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b798daf216454a4598437947928803d1" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:26.543963 15443 raft_consensus.cc:2781] T 1461cdeda5664afd9c4c2b45199179b6 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:26.551895 15234 leader_election.cc:302] T b798daf216454a4598437947928803d1 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae; no voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224
I20260810 08:00:26.553000 15447 raft_consensus.cc:3037] T b798daf216454a4598437947928803d1 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:26.545421 15478 ts_tablet_manager.cc:1905] T dc3ea02864d040c5bcde38b1a5609676 P 3fc5e568f3724f158b616c303efb9eae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:26.573283 15490 tablet_bootstrap.cc:492] T 114d3eac994f466bb1787e066de05e2c P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:26.573767 15490 ts_tablet_manager.cc:1397] T 114d3eac994f466bb1787e066de05e2c P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.145s user 0.012s sys 0.002s
W20260810 08:00:26.574820 15079 consensus_peers.cc:487] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 9d7219686f304f62a54acc7497cbbd07 -> Peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Couldn't send request to peer a91b27762182454baada71e91ceab224. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aeb7c1c4bb1f45f9b46fe5bd78e96a20. This is attempt 1: this message will repeat every 5th retry.
I20260810 08:00:26.575939 15490 raft_consensus.cc:348] T 114d3eac994f466bb1787e066de05e2c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:26.576563 15490 raft_consensus.cc:374] T 114d3eac994f466bb1787e066de05e2c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:26.576781 15490 raft_consensus.cc:729] T 114d3eac994f466bb1787e066de05e2c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:26.567549 15443 raft_consensus.cc:482] T 1461cdeda5664afd9c4c2b45199179b6 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:26.579439 15443 raft_consensus.cc:3037] T 1461cdeda5664afd9c4c2b45199179b6 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:26.596752 15437 tablet_bootstrap.cc:492] T 472fbc9204754317aab059a4844d7de1 P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:26.597410 15437 ts_tablet_manager.cc:1397] T 472fbc9204754317aab059a4844d7de1 P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.504s user 0.017s sys 0.002s
I20260810 08:00:26.597688 15439 tablet_bootstrap.cc:492] T a246b8cad99b47a9a856f6aed70bdfc8 P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:26.583567 15482 tablet_replica.cc:330] T 9ddae63b29c34e1397c5765e52e1478f P a91b27762182454baada71e91ceab224: stopping tablet replica
I20260810 08:00:26.600792 15482 raft_consensus.cc:2229] T 9ddae63b29c34e1397c5765e52e1478f P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:26.601203 15482 raft_consensus.cc:2258] T 9ddae63b29c34e1397c5765e52e1478f P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:26.602608 15479 ts_tablet_manager.cc:1918] T abdd2b119a4b448dacdd2d87da34832f P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:26.602720 15482 ts_tablet_manager.cc:1905] T 9ddae63b29c34e1397c5765e52e1478f P a91b27762182454baada71e91ceab224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:26.602972 15479 log.cc:1196] T abdd2b119a4b448dacdd2d87da34832f P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/abdd2b119a4b448dacdd2d87da34832f
I20260810 08:00:26.603561 15479 ts_tablet_manager.cc:1939] T abdd2b119a4b448dacdd2d87da34832f P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:26.608217 15479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43002 (request call id 21) took 2403 ms. Trace:
I20260810 08:00:26.608714 15443 raft_consensus.cc:504] T 1461cdeda5664afd9c4c2b45199179b6 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:26.610136 14690 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_7beb9923b5d5438984557b4df7910362" } new_table_name: "default.Table_f387e847280a4f048905e15c5003f58c" modify_external_catalogs: true
I20260810 08:00:26.609701 15490 consensus_queue.cc:260] T 114d3eac994f466bb1787e066de05e2c P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:26.612958 15147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1461cdeda5664afd9c4c2b45199179b6" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07"
I20260810 08:00:26.613517 15147 raft_consensus.cc:3037] T 1461cdeda5664afd9c4c2b45199179b6 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:26.615659 15490 ts_tablet_manager.cc:1428] T 114d3eac994f466bb1787e066de05e2c P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.042s user 0.004s sys 0.001s
I20260810 08:00:26.615525 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1461cdeda5664afd9c4c2b45199179b6" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae"
I20260810 08:00:26.616544 15490 tablet_bootstrap.cc:492] T 35b6da60a5544d7e9b657a5f1e6f90ff P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
W20260810 08:00:26.617064 14918 leader_election.cc:341] T 1461cdeda5664afd9c4c2b45199179b6 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:26.621186 15147 raft_consensus.cc:2454] T 1461cdeda5664afd9c4c2b45199179b6 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 1.
I20260810 08:00:26.622392 14922 leader_election.cc:302] T 1461cdeda5664afd9c4c2b45199179b6 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224; no voters: 3fc5e568f3724f158b616c303efb9eae
I20260810 08:00:26.608551 15479 rpcz_store.cc:276] 0810 08:00:24.204875 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:24.205037 (+ 162us) service_pool.cc:227] Handling call
0810 08:00:26.490064 (+2285027us) ts_tablet_manager.cc:1145] Deleting tablet abdd2b119a4b448dacdd2d87da34832f
0810 08:00:26.490139 (+ 75us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:26.586601 (+ 96462us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:26.602395 (+ 15794us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:26.608149 (+ 5754us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2261007}
I20260810 08:00:26.628875 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet abdd2b119a4b448dacdd2d87da34832f (table default.Table_3a39902b550c4390a56429d0ef94860e [id=cee29bc23d044a4bae553990b0d3c32c]) successfully deleted
I20260810 08:00:26.630439 15479 tablet_replica.cc:330] T 5e0e4f1c76f843ea9000ed25acec73d3 P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:26.632004 15479 raft_consensus.cc:2229] T 5e0e4f1c76f843ea9000ed25acec73d3 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:26.632436 15479 raft_consensus.cc:2258] T 5e0e4f1c76f843ea9000ed25acec73d3 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:26.634058 15479 ts_tablet_manager.cc:1905] T 5e0e4f1c76f843ea9000ed25acec73d3 P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:26.635962 15447 raft_consensus.cc:2726] T b798daf216454a4598437947928803d1 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:26.637593 14689 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet 35b6da60a5544d7e9b657a5f1e6f90ff
I20260810 08:00:26.640302 15122 tablet_service.cc:1514] Processing DeleteTablet for tablet 35b6da60a5544d7e9b657a5f1e6f90ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:26 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:26.641816 15278 tablet_service.cc:1514] Processing DeleteTablet for tablet 35b6da60a5544d7e9b657a5f1e6f90ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:26 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:26.599848 15437 raft_consensus.cc:348] T 472fbc9204754317aab059a4844d7de1 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:26.642441 15437 raft_consensus.cc:374] T 472fbc9204754317aab059a4844d7de1 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:26.642714 15437 raft_consensus.cc:729] T 472fbc9204754317aab059a4844d7de1 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:26.643348 15437 consensus_queue.cc:260] T 472fbc9204754317aab059a4844d7de1 P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:26.644449 15443 leader_election.cc:288] T 1461cdeda5664afd9c4c2b45199179b6 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Requested vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:26.644963 15443 raft_consensus.cc:2781] T 1461cdeda5664afd9c4c2b45199179b6 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 08:00:26.645363 15443 raft_consensus.cc:686] T 1461cdeda5664afd9c4c2b45199179b6 P a91b27762182454baada71e91ceab224 [term 1 LEADER]: Becoming Leader. State: Replica: a91b27762182454baada71e91ceab224, State: Running, Role: LEADER
W20260810 08:00:26.652576 15076 consensus_peers.cc:487] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 9d7219686f304f62a54acc7497cbbd07 -> Peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Couldn't send request to peer 3fc5e568f3724f158b616c303efb9eae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aeb7c1c4bb1f45f9b46fe5bd78e96a20. This is attempt 1: this message will repeat every 5th retry.
I20260810 08:00:26.655827 15490 tablet_bootstrap.cc:654] T 35b6da60a5544d7e9b657a5f1e6f90ff P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:26.667646 15439 tablet_bootstrap.cc:654] T a246b8cad99b47a9a856f6aed70bdfc8 P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
W20260810 08:00:26.674665 14692 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5a22a48e2ba843b3a32a8900ab937ea0 consensus state has no leader
I20260810 08:00:26.680630 15479 ts_tablet_manager.cc:1918] T 5e0e4f1c76f843ea9000ed25acec73d3 P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:26.683696 15479 log.cc:1196] T 5e0e4f1c76f843ea9000ed25acec73d3 P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/5e0e4f1c76f843ea9000ed25acec73d3
I20260810 08:00:26.685153 15479 ts_tablet_manager.cc:1939] T 5e0e4f1c76f843ea9000ed25acec73d3 P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:26.687638 15479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43002 (request call id 24) took 2219 ms. Trace:
I20260810 08:00:26.688133 15478 ts_tablet_manager.cc:1918] T dc3ea02864d040c5bcde38b1a5609676 P 3fc5e568f3724f158b616c303efb9eae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 08:00:26.688501 15478 log.cc:1196] T dc3ea02864d040c5bcde38b1a5609676 P 3fc5e568f3724f158b616c303efb9eae: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal/wals/dc3ea02864d040c5bcde38b1a5609676
I20260810 08:00:26.689211 15478 ts_tablet_manager.cc:1939] T dc3ea02864d040c5bcde38b1a5609676 P 3fc5e568f3724f158b616c303efb9eae: Deleting consensus metadata
I20260810 08:00:26.689713 14694 catalog_manager.cc:5777] Not sending DeleteTablet requests; no consensus state for tablet f4cafec906564957b01f2227283baccb
I20260810 08:00:26.690114 15479 rpcz_store.cc:276] 0810 08:00:24.468506 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:24.468631 (+ 125us) service_pool.cc:227] Handling call
0810 08:00:26.630280 (+2161649us) ts_tablet_manager.cc:1145] Deleting tablet 5e0e4f1c76f843ea9000ed25acec73d3
0810 08:00:26.630327 (+ 47us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:26.659548 (+ 29221us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:26.680407 (+ 20859us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:26.687584 (+ 7177us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2161082}
I20260810 08:00:26.691977 15490 tablet_bootstrap.cc:492] T 35b6da60a5544d7e9b657a5f1e6f90ff P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:26.692405 15490 ts_tablet_manager.cc:1397] T 35b6da60a5544d7e9b657a5f1e6f90ff P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.076s user 0.005s sys 0.005s
I20260810 08:00:26.692457 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet 5e0e4f1c76f843ea9000ed25acec73d3 (table default.Table_43a063da4167472198ae9533501169eb [id=9cbd826186904ef687327cd9e02fdc1a]) successfully deleted
I20260810 08:00:26.693001 15479 tablet_replica.cc:330] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:26.693986 15479 raft_consensus.cc:2229] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 9d7219686f304f62a54acc7497cbbd07 [term 1 LEADER]: Raft consensus shutting down.
I20260810 08:00:26.695379 15490 raft_consensus.cc:348] T 35b6da60a5544d7e9b657a5f1e6f90ff P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:26.696156 15490 raft_consensus.cc:374] T 35b6da60a5544d7e9b657a5f1e6f90ff P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:26.696416 15490 raft_consensus.cc:729] T 35b6da60a5544d7e9b657a5f1e6f90ff P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:26.696575 15479 pending_rounds.cc:62] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 9d7219686f304f62a54acc7497cbbd07: Trying to abort 1 pending ops.
I20260810 08:00:26.697134 15479 pending_rounds.cc:68] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 9d7219686f304f62a54acc7497cbbd07: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316884788947664896 op_type: NO_OP noop_request { }
W20260810 08:00:26.698019 14679 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 5a22a48e2ba843b3a32a8900ab937ea0 (table default.Table_5a6ab9b9acc9408c8811f77acaca9a0c [id=ac3a5787385940bca8030d7bac893a27], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5a22a48e2ba843b3a32a8900ab937ea0 consensus state has no leader
I20260810 08:00:26.699240 15439 tablet_bootstrap.cc:492] T a246b8cad99b47a9a856f6aed70bdfc8 P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:26.699647 15439 ts_tablet_manager.cc:1397] T a246b8cad99b47a9a856f6aed70bdfc8 P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.102s user 0.007s sys 0.002s
I20260810 08:00:26.664669 15443 consensus_queue.cc:237] T 1461cdeda5664afd9c4c2b45199179b6 P a91b27762182454baada71e91ceab224 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
W20260810 08:00:26.704778 14922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.194:36265, user_credentials={real_user=slave}} blocked reactor thread for 81901.1us
I20260810 08:00:26.714634 15479 raft_consensus.cc:2866] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 9d7219686f304f62a54acc7497cbbd07 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260810 08:00:26.710165 15490 consensus_queue.cc:260] T 35b6da60a5544d7e9b657a5f1e6f90ff P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:26.719029 15490 ts_tablet_manager.cc:1428] T 35b6da60a5544d7e9b657a5f1e6f90ff P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.026s user 0.004s sys 0.003s
I20260810 08:00:26.726874 15490 tablet_bootstrap.cc:492] T a1c4f8f514c94128aedd07bd5efdc3be P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:26.715006 15479 raft_consensus.cc:2258] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:26.731880 14965 tablet_service.cc:1514] Processing DeleteTablet for tablet 35b6da60a5544d7e9b657a5f1e6f90ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:26 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:26.735100 15439 raft_consensus.cc:348] T a246b8cad99b47a9a856f6aed70bdfc8 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:26.735785 15439 raft_consensus.cc:374] T a246b8cad99b47a9a856f6aed70bdfc8 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:26.736009 15439 raft_consensus.cc:729] T a246b8cad99b47a9a856f6aed70bdfc8 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:26.740831 15439 consensus_queue.cc:260] T a246b8cad99b47a9a856f6aed70bdfc8 P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:26.741955 15479 ts_tablet_manager.cc:1905] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:26.746912 15490 tablet_bootstrap.cc:654] T a1c4f8f514c94128aedd07bd5efdc3be P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:26.747633 15573 raft_consensus.cc:482] T 35b6da60a5544d7e9b657a5f1e6f90ff P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:26.748677 15573 raft_consensus.cc:504] T 35b6da60a5544d7e9b657a5f1e6f90ff P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:26.749981 15437 ts_tablet_manager.cc:1428] T 472fbc9204754317aab059a4844d7de1 P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.152s user 0.003s sys 0.007s
I20260810 08:00:26.750589 15573 leader_election.cc:288] T 35b6da60a5544d7e9b657a5f1e6f90ff P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:26.750871 15437 tablet_bootstrap.cc:492] T 35b6da60a5544d7e9b657a5f1e6f90ff P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:26.753247 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b6da60a5544d7e9b657a5f1e6f90ff" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:26.753759 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b6da60a5544d7e9b657a5f1e6f90ff" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
W20260810 08:00:26.755250 15079 leader_election.cc:341] T 35b6da60a5544d7e9b657a5f1e6f90ff P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
W20260810 08:00:26.756261 15076 leader_election.cc:341] T 35b6da60a5544d7e9b657a5f1e6f90ff P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:26.756664 15076 leader_election.cc:302] T 35b6da60a5544d7e9b657a5f1e6f90ff P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:26.757433 15437 tablet_bootstrap.cc:654] T 35b6da60a5544d7e9b657a5f1e6f90ff P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:26.757838 15573 raft_consensus.cc:2726] T 35b6da60a5544d7e9b657a5f1e6f90ff P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:26.758728 14694 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_3099152ccd3b4d5a8b558ee964aefaf8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:26.763361 14688 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet 0c152aab7e244afd857534869b13815c
I20260810 08:00:26.764853 14688 catalog_manager.cc:6565] ReplaceTablet: tablet 0c152aab7e244afd857534869b13815c deleted and replaced by tablet 8a8454f693494ab3b82f5b85c2d62bb2
W20260810 08:00:26.765527 14694 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3099152ccd3b4d5a8b558ee964aefaf8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:26.768584 15278 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c152aab7e244afd857534869b13815c with delete_type TABLET_DATA_DELETED (replaced by tablet 8a8454f693494ab3b82f5b85c2d62bb2) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:26.770648 15122 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c152aab7e244afd857534869b13815c with delete_type TABLET_DATA_DELETED (replaced by tablet 8a8454f693494ab3b82f5b85c2d62bb2) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:26.773272 15439 ts_tablet_manager.cc:1428] T a246b8cad99b47a9a856f6aed70bdfc8 P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.073s user 0.006s sys 0.001s
I20260810 08:00:26.782557 15439 tablet_bootstrap.cc:492] T 472fbc9204754317aab059a4844d7de1 P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:26.787287 15447 raft_consensus.cc:482] T abdd2b119a4b448dacdd2d87da34832f P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:26.787743 15447 raft_consensus.cc:504] T abdd2b119a4b448dacdd2d87da34832f P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
W20260810 08:00:26.792444 15234 leader_election.cc:341] T abdd2b119a4b448dacdd2d87da34832f P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Not found: Tablet not found: abdd2b119a4b448dacdd2d87da34832f
I20260810 08:00:26.789599 15447 leader_election.cc:288] T abdd2b119a4b448dacdd2d87da34832f P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265), a91b27762182454baada71e91ceab224 (127.14.57.193:43601)
I20260810 08:00:26.793819 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 39) took 1261 ms. Trace:
I20260810 08:00:26.794278 15478 rpcz_store.cc:276] 0810 08:00:25.532982 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:25.533211 (+ 229us) service_pool.cc:227] Handling call
0810 08:00:26.532601 (+999390us) ts_tablet_manager.cc:1145] Deleting tablet dc3ea02864d040c5bcde38b1a5609676
0810 08:00:26.532642 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:26.638366 (+105724us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:26.687915 (+ 49549us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:26.793722 (+105807us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":998611}
I20260810 08:00:26.796214 15490 tablet_bootstrap.cc:492] T a1c4f8f514c94128aedd07bd5efdc3be P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
W20260810 08:00:26.802026 15234 leader_election.cc:341] T abdd2b119a4b448dacdd2d87da34832f P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Not found: Tablet not found: abdd2b119a4b448dacdd2d87da34832f
I20260810 08:00:26.802471 15234 leader_election.cc:302] T abdd2b119a4b448dacdd2d87da34832f P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae; no voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224
W20260810 08:00:26.802743 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet 9ddae63b29c34e1397c5765e52e1478f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ddae63b29c34e1397c5765e52e1478f already in progress: creating tablet
I20260810 08:00:26.802865 15490 ts_tablet_manager.cc:1397] T a1c4f8f514c94128aedd07bd5efdc3be P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.076s user 0.008s sys 0.002s
I20260810 08:00:26.803550 15447 raft_consensus.cc:2726] T abdd2b119a4b448dacdd2d87da34832f P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:26.807531 15490 raft_consensus.cc:348] T a1c4f8f514c94128aedd07bd5efdc3be P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:26.808193 15490 raft_consensus.cc:374] T a1c4f8f514c94128aedd07bd5efdc3be P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:26.808427 15490 raft_consensus.cc:729] T a1c4f8f514c94128aedd07bd5efdc3be P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:26.772492 14965 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c152aab7e244afd857534869b13815c with delete_type TABLET_DATA_DELETED (replaced by tablet 8a8454f693494ab3b82f5b85c2d62bb2) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:26.800618 15588 raft_consensus.cc:482] T 472fbc9204754317aab059a4844d7de1 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:26.813551 15588 raft_consensus.cc:504] T 472fbc9204754317aab059a4844d7de1 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:26.822919 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472fbc9204754317aab059a4844d7de1" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:26.823086 15588 leader_election.cc:288] T 472fbc9204754317aab059a4844d7de1 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
W20260810 08:00:26.824260 14918 leader_election.cc:341] T 472fbc9204754317aab059a4844d7de1 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:26.826042 15439 tablet_bootstrap.cc:654] T 472fbc9204754317aab059a4844d7de1 P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:26.825006 15490 consensus_queue.cc:260] T a1c4f8f514c94128aedd07bd5efdc3be P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:26.832307 15478 tablet_replica.cc:330] T abdd2b119a4b448dacdd2d87da34832f P 3fc5e568f3724f158b616c303efb9eae: stopping tablet replica
I20260810 08:00:26.811075 15482 ts_tablet_manager.cc:1918] T 9ddae63b29c34e1397c5765e52e1478f P a91b27762182454baada71e91ceab224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:26.832211 15147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472fbc9204754317aab059a4844d7de1" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:26.832828 15147 raft_consensus.cc:2454] T 472fbc9204754317aab059a4844d7de1 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 0.
I20260810 08:00:26.834481 14922 leader_election.cc:302] T 472fbc9204754317aab059a4844d7de1 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224; no voters: 3fc5e568f3724f158b616c303efb9eae
I20260810 08:00:26.835399 15490 ts_tablet_manager.cc:1428] T a1c4f8f514c94128aedd07bd5efdc3be P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.032s user 0.005s sys 0.000s
I20260810 08:00:26.835708 15588 raft_consensus.cc:2781] T 472fbc9204754317aab059a4844d7de1 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:26.836103 15588 raft_consensus.cc:482] T 472fbc9204754317aab059a4844d7de1 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:26.836467 15588 raft_consensus.cc:3037] T 472fbc9204754317aab059a4844d7de1 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:26.849640 15588 raft_consensus.cc:504] T 472fbc9204754317aab059a4844d7de1 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:26.851455 15490 tablet_bootstrap.cc:492] T 5a22a48e2ba843b3a32a8900ab937ea0 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:26.851722 14687 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_b7ee9352799b44f4b7bbf48355b3af11" } modify_external_catalogs: true
I20260810 08:00:26.853003 14687 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_b7ee9352799b44f4b7bbf48355b3af11" } modify_external_catalogs: true
I20260810 08:00:26.861449 14693 master_service.cc:946] ReplaceTablet: received request to replace tablet 728f8f80b3bb4b10871d8b400f25377b from {username='slave'} at 127.0.0.1:42300
I20260810 08:00:26.862890 15147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472fbc9204754317aab059a4844d7de1" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07"
I20260810 08:00:26.863531 15147 raft_consensus.cc:3037] T 472fbc9204754317aab059a4844d7de1 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:26.870980 15147 raft_consensus.cc:2454] T 472fbc9204754317aab059a4844d7de1 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 1.
I20260810 08:00:26.872176 14922 leader_election.cc:302] T 472fbc9204754317aab059a4844d7de1 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224; no voters:
I20260810 08:00:26.872958 15443 raft_consensus.cc:2781] T 472fbc9204754317aab059a4844d7de1 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 08:00:26.873412 15443 raft_consensus.cc:686] T 472fbc9204754317aab059a4844d7de1 P a91b27762182454baada71e91ceab224 [term 1 LEADER]: Becoming Leader. State: Replica: a91b27762182454baada71e91ceab224, State: Running, Role: LEADER
I20260810 08:00:26.874176 15443 consensus_queue.cc:237] T 472fbc9204754317aab059a4844d7de1 P a91b27762182454baada71e91ceab224 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:26.879065 14689 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_eb8d1ba0b9fc4c789eb3a43b70ed093b" } new_table_name: "default.Table_89fe7ad723964551909b73ff6b72b9af" modify_external_catalogs: true
I20260810 08:00:26.879447 15490 tablet_bootstrap.cc:654] T 5a22a48e2ba843b3a32a8900ab937ea0 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:26.881023 15588 leader_election.cc:288] T 472fbc9204754317aab059a4844d7de1 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Requested vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:26.882318 15479 ts_tablet_manager.cc:1918] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 08:00:26.882694 15479 log.cc:1196] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/aeb7c1c4bb1f45f9b46fe5bd78e96a20
I20260810 08:00:26.885110 15479 ts_tablet_manager.cc:1939] T aeb7c1c4bb1f45f9b46fe5bd78e96a20 P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:26.887144 15479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43002 (request call id 27) took 2224 ms. Trace:
I20260810 08:00:26.887444 15479 rpcz_store.cc:276] 0810 08:00:24.663021 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:24.663183 (+ 162us) service_pool.cc:227] Handling call
0810 08:00:26.692450 (+2029267us) ts_tablet_manager.cc:1145] Deleting tablet aeb7c1c4bb1f45f9b46fe5bd78e96a20
0810 08:00:26.692489 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:26.817176 (+124687us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:26.870918 (+ 53742us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:26.887087 (+ 16169us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2027333}
I20260810 08:00:26.888643 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet aeb7c1c4bb1f45f9b46fe5bd78e96a20 (table default.Table_9b0d38b98f3841828a00a23dda57ae6e [id=40cf37d4ad57413fa64ee743a5c160c3]) successfully deleted
I20260810 08:00:26.899648 15490 tablet_bootstrap.cc:492] T 5a22a48e2ba843b3a32a8900ab937ea0 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:26.900103 15490 ts_tablet_manager.cc:1397] T 5a22a48e2ba843b3a32a8900ab937ea0 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.049s user 0.009s sys 0.000s
I20260810 08:00:26.832862 15482 log.cc:1196] T 9ddae63b29c34e1397c5765e52e1478f P a91b27762182454baada71e91ceab224: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal/wals/9ddae63b29c34e1397c5765e52e1478f
I20260810 08:00:26.857931 15439 tablet_bootstrap.cc:492] T 472fbc9204754317aab059a4844d7de1 P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:26.862438 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472fbc9204754317aab059a4844d7de1" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae"
I20260810 08:00:26.903200 15479 tablet_replica.cc:330] T a246b8cad99b47a9a856f6aed70bdfc8 P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
W20260810 08:00:26.903445 14918 leader_election.cc:341] T 472fbc9204754317aab059a4844d7de1 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:26.903928 15479 raft_consensus.cc:2229] T a246b8cad99b47a9a856f6aed70bdfc8 P 9d7219686f304f62a54acc7497cbbd07 [term 1 LEADER]: Raft consensus shutting down.
I20260810 08:00:26.904402 15439 ts_tablet_manager.cc:1397] T 472fbc9204754317aab059a4844d7de1 P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.122s user 0.003s sys 0.008s
I20260810 08:00:26.904569 15479 pending_rounds.cc:62] T a246b8cad99b47a9a856f6aed70bdfc8 P 9d7219686f304f62a54acc7497cbbd07: Trying to abort 1 pending ops.
I20260810 08:00:26.904783 15479 pending_rounds.cc:68] T a246b8cad99b47a9a856f6aed70bdfc8 P 9d7219686f304f62a54acc7497cbbd07: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316884793417191424 op_type: NO_OP noop_request { }
I20260810 08:00:26.905140 15479 raft_consensus.cc:2866] T a246b8cad99b47a9a856f6aed70bdfc8 P 9d7219686f304f62a54acc7497cbbd07 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260810 08:00:26.907949 15490 raft_consensus.cc:348] T 5a22a48e2ba843b3a32a8900ab937ea0 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:26.908589 15490 raft_consensus.cc:374] T 5a22a48e2ba843b3a32a8900ab937ea0 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:26.908428 15439 raft_consensus.cc:348] T 472fbc9204754317aab059a4844d7de1 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:26.908838 15490 raft_consensus.cc:729] T 5a22a48e2ba843b3a32a8900ab937ea0 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:26.909111 15439 raft_consensus.cc:374] T 472fbc9204754317aab059a4844d7de1 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:26.909363 15439 raft_consensus.cc:729] T 472fbc9204754317aab059a4844d7de1 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:26.905447 15479 raft_consensus.cc:2258] T a246b8cad99b47a9a856f6aed70bdfc8 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:26.910068 15439 consensus_queue.cc:260] T 472fbc9204754317aab059a4844d7de1 P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:26.914343 15490 consensus_queue.cc:260] T 5a22a48e2ba843b3a32a8900ab937ea0 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:26.915658 15439 ts_tablet_manager.cc:1428] T 472fbc9204754317aab059a4844d7de1 P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260810 08:00:26.916041 15482 ts_tablet_manager.cc:1939] T 9ddae63b29c34e1397c5765e52e1478f P a91b27762182454baada71e91ceab224: Deleting consensus metadata
I20260810 08:00:26.916644 15479 ts_tablet_manager.cc:1905] T a246b8cad99b47a9a856f6aed70bdfc8 P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:26.919096 14681 catalog_manager.cc:4841] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): tablet 9ddae63b29c34e1397c5765e52e1478f (table default.Table_0e30c621046242609c9f6fc73b16d0e7 [id=e9608d66a9964bc6aafdf84f574fba02]) successfully deleted
I20260810 08:00:26.920214 15478 raft_consensus.cc:2229] T abdd2b119a4b448dacdd2d87da34832f P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:26.920696 15478 raft_consensus.cc:2258] T abdd2b119a4b448dacdd2d87da34832f P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:26.925753 15490 ts_tablet_manager.cc:1428] T 5a22a48e2ba843b3a32a8900ab937ea0 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.025s user 0.007s sys 0.001s
I20260810 08:00:26.932021 15490 tablet_bootstrap.cc:492] T e46648198f5343f7a65079887e5df159 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:26.935289 14695 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_1e20f010e84746b0a487c82539f9ab8a" } modify_external_catalogs: true
I20260810 08:00:26.935868 14695 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_1e20f010e84746b0a487c82539f9ab8a" } modify_external_catalogs: true
I20260810 08:00:26.945267 15437 tablet_bootstrap.cc:492] T 35b6da60a5544d7e9b657a5f1e6f90ff P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:26.945775 15437 ts_tablet_manager.cc:1397] T 35b6da60a5544d7e9b657a5f1e6f90ff P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.195s user 0.010s sys 0.001s
I20260810 08:00:26.952572 15490 tablet_bootstrap.cc:654] T e46648198f5343f7a65079887e5df159 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
W20260810 08:00:26.829295 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet 37d8a37a16f34a0285c43c0d71b5f072 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37d8a37a16f34a0285c43c0d71b5f072 already in progress: creating tablet
W20260810 08:00:26.959952 14678 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.14.57.195:37729, user_credentials={real_user=slave}} blocked reactor thread for 130687us
W20260810 08:00:26.961450 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet 472fbc9204754317aab059a4844d7de1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 472fbc9204754317aab059a4844d7de1 already in progress: creating tablet
I20260810 08:00:26.982112 14686 master_service.cc:946] ReplaceTablet: received request to replace tablet fbf62f4eab8b49f282df3ab4d95d886f from {username='slave'} at 127.0.0.1:42300
I20260810 08:00:26.986690 15588 consensus_queue.cc:1035] T b798daf216454a4598437947928803d1 P a91b27762182454baada71e91ceab224 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260810 08:00:26.995004 14690 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d0761b090d8147b8a5537b0ed51108a9 has no consensus state
I20260810 08:00:27.008368 14688 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_b7a38b32fda646888162ee796bb97bbf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:27.010674 14688 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7a38b32fda646888162ee796bb97bbf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:27.011482 15437 raft_consensus.cc:348] T 35b6da60a5544d7e9b657a5f1e6f90ff P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:27.012336 15437 raft_consensus.cc:374] T 35b6da60a5544d7e9b657a5f1e6f90ff P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:27.012596 15437 raft_consensus.cc:729] T 35b6da60a5544d7e9b657a5f1e6f90ff P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:27.018289 15478 ts_tablet_manager.cc:1905] T abdd2b119a4b448dacdd2d87da34832f P 3fc5e568f3724f158b616c303efb9eae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:27.022126 15278 tablet_service.cc:1514] Processing DeleteTablet for tablet a246b8cad99b47a9a856f6aed70bdfc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:24 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:27.023685 15439 tablet_bootstrap.cc:492] T 1461cdeda5664afd9c4c2b45199179b6 P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:27.028247 15482 tablet_replica.cc:330] T 472fbc9204754317aab059a4844d7de1 P a91b27762182454baada71e91ceab224: stopping tablet replica
I20260810 08:00:27.028957 15482 raft_consensus.cc:2229] T 472fbc9204754317aab059a4844d7de1 P a91b27762182454baada71e91ceab224 [term 1 LEADER]: Raft consensus shutting down.
I20260810 08:00:27.030037 15482 pending_rounds.cc:62] T 472fbc9204754317aab059a4844d7de1 P a91b27762182454baada71e91ceab224: Trying to abort 1 pending ops.
I20260810 08:00:27.030252 15482 pending_rounds.cc:68] T 472fbc9204754317aab059a4844d7de1 P a91b27762182454baada71e91ceab224: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316884794819854336 op_type: NO_OP noop_request { }
I20260810 08:00:27.030747 15482 raft_consensus.cc:2866] T 472fbc9204754317aab059a4844d7de1 P a91b27762182454baada71e91ceab224 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260810 08:00:27.031076 15482 raft_consensus.cc:2258] T 472fbc9204754317aab059a4844d7de1 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:27.009507 15479 ts_tablet_manager.cc:1918] T a246b8cad99b47a9a856f6aed70bdfc8 P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 08:00:27.034937 15443 consensus_queue.cc:1035] T b798daf216454a4598437947928803d1 P a91b27762182454baada71e91ceab224 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 08:00:27.035521 15479 log.cc:1196] T a246b8cad99b47a9a856f6aed70bdfc8 P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/a246b8cad99b47a9a856f6aed70bdfc8
I20260810 08:00:27.036237 15479 ts_tablet_manager.cc:1939] T a246b8cad99b47a9a856f6aed70bdfc8 P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:27.042603 15479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43002 (request call id 28) took 2129 ms. Trace:
I20260810 08:00:27.042930 15479 rpcz_store.cc:276] 0810 08:00:24.913057 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:24.913216 (+ 159us) service_pool.cc:227] Handling call
0810 08:00:26.903049 (+1989833us) ts_tablet_manager.cc:1145] Deleting tablet a246b8cad99b47a9a856f6aed70bdfc8
0810 08:00:26.903092 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:26.962909 (+ 59817us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:27.009176 (+ 46267us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:27.042536 (+ 33360us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1951541}
I20260810 08:00:27.044077 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet a246b8cad99b47a9a856f6aed70bdfc8 (table default.Table_491d9caa5f6d4e3cb934a5d0582b519c [id=d106e75991f54f60abbe60b0495cb8e8]) successfully deleted
I20260810 08:00:27.052049 15479 tablet_replica.cc:330] T 92bbad33a92643e3be3899ace5e29f52 P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:27.052809 15479 raft_consensus.cc:2229] T 92bbad33a92643e3be3899ace5e29f52 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:27.053256 15479 raft_consensus.cc:2258] T 92bbad33a92643e3be3899ace5e29f52 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:27.055119 15479 ts_tablet_manager.cc:1905] T 92bbad33a92643e3be3899ace5e29f52 P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:27.060005 15278 tablet_service.cc:1514] Processing DeleteTablet for tablet 472fbc9204754317aab059a4844d7de1 with delete_type TABLET_DATA_DELETED (replaced by tablet b5a0e0af742340b2945be2102768b307) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:27.062953 15447 raft_consensus.cc:482] T 472fbc9204754317aab059a4844d7de1 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:27.063410 15447 raft_consensus.cc:504] T 472fbc9204754317aab059a4844d7de1 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:27.065270 15447 leader_election.cc:288] T 472fbc9204754317aab059a4844d7de1 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
W20260810 08:00:27.074908 14689 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a1c4f8f514c94128aedd07bd5efdc3be consensus state has no leader
W20260810 08:00:27.087174 14681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.57.252:33819: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20260810 08:00:27.087836 15439 tablet_bootstrap.cc:654] T 1461cdeda5664afd9c4c2b45199179b6 P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:27.095031 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472fbc9204754317aab059a4844d7de1" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:27.097782 15479 ts_tablet_manager.cc:1918] T 92bbad33a92643e3be3899ace5e29f52 P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20260810 08:00:27.098073 15234 leader_election.cc:341] T 472fbc9204754317aab059a4844d7de1 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:27.013861 15437 consensus_queue.cc:260] T 35b6da60a5544d7e9b657a5f1e6f90ff P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:27.098601 15146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472fbc9204754317aab059a4844d7de1" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:27.102171 15146 raft_consensus.cc:2379] T 472fbc9204754317aab059a4844d7de1 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fc5e568f3724f158b616c303efb9eae in current term 1: Already voted for candidate a91b27762182454baada71e91ceab224 in this term.
I20260810 08:00:27.102377 15437 ts_tablet_manager.cc:1428] T 35b6da60a5544d7e9b657a5f1e6f90ff P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.156s user 0.006s sys 0.000s
I20260810 08:00:27.103184 15479 log.cc:1196] T 92bbad33a92643e3be3899ace5e29f52 P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/92bbad33a92643e3be3899ace5e29f52
I20260810 08:00:27.105172 15479 ts_tablet_manager.cc:1939] T 92bbad33a92643e3be3899ace5e29f52 P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:27.107062 14689 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_d423114f2dce44ffaf0fad0c40c5b985"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:27.109094 14689 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d423114f2dce44ffaf0fad0c40c5b985 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:27.113273 14694 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_f3dd0fbbcd664605b91fb032e4877812"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:27.113730 15234 leader_election.cc:302] T 472fbc9204754317aab059a4844d7de1 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae; no voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224
W20260810 08:00:27.115034 14694 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3dd0fbbcd664605b91fb032e4877812 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:27.120958 15482 ts_tablet_manager.cc:1905] T 472fbc9204754317aab059a4844d7de1 P a91b27762182454baada71e91ceab224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:27.105641 15490 tablet_bootstrap.cc:492] T e46648198f5343f7a65079887e5df159 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:27.122049 15490 ts_tablet_manager.cc:1397] T e46648198f5343f7a65079887e5df159 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.190s user 0.014s sys 0.018s
I20260810 08:00:27.124289 15490 raft_consensus.cc:348] T e46648198f5343f7a65079887e5df159 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:27.124984 15490 raft_consensus.cc:374] T e46648198f5343f7a65079887e5df159 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:27.125205 15490 raft_consensus.cc:729] T e46648198f5343f7a65079887e5df159 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:27.125766 15437 tablet_bootstrap.cc:492] T a1c4f8f514c94128aedd07bd5efdc3be P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:27.125808 15490 consensus_queue.cc:260] T e46648198f5343f7a65079887e5df159 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:27.128623 15479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43002 (request call id 31) took 2043 ms. Trace:
I20260810 08:00:27.128925 15479 rpcz_store.cc:276] 0810 08:00:25.085696 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:25.085834 (+ 138us) service_pool.cc:227] Handling call
0810 08:00:27.051888 (+1966054us) ts_tablet_manager.cc:1145] Deleting tablet 92bbad33a92643e3be3899ace5e29f52
0810 08:00:27.051934 (+ 46us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:27.065340 (+ 13406us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:27.097541 (+ 32201us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:27.128563 (+ 31022us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1965458}
I20260810 08:00:27.129316 15447 raft_consensus.cc:3037] T 472fbc9204754317aab059a4844d7de1 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:27.136544 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet 92bbad33a92643e3be3899ace5e29f52 (table default.Table_7f3a4e19ded64b3783a00b3d4c6839c3 [id=6c6afcf578534997878b98feb9574e2d]) successfully deleted
I20260810 08:00:27.136878 15479 tablet_replica.cc:330] T 2faab63c5023431c98e7b01ce61e04c2 P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:27.136991 15447 raft_consensus.cc:2726] T 472fbc9204754317aab059a4844d7de1 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:27.137540 15479 raft_consensus.cc:2229] T 2faab63c5023431c98e7b01ce61e04c2 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:27.137974 15479 raft_consensus.cc:2258] T 2faab63c5023431c98e7b01ce61e04c2 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:27.151768 15479 ts_tablet_manager.cc:1905] T 2faab63c5023431c98e7b01ce61e04c2 P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:27.156445 15437 tablet_bootstrap.cc:654] T a1c4f8f514c94128aedd07bd5efdc3be P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
W20260810 08:00:27.165369 15076 consensus_peers.cc:487] T 9ddae63b29c34e1397c5765e52e1478f P 9d7219686f304f62a54acc7497cbbd07 -> Peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Couldn't send request to peer 3fc5e568f3724f158b616c303efb9eae. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260810 08:00:27.179507 15439 tablet_bootstrap.cc:492] T 1461cdeda5664afd9c4c2b45199179b6 P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:27.179807 15490 ts_tablet_manager.cc:1428] T e46648198f5343f7a65079887e5df159 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.057s user 0.008s sys 0.002s
I20260810 08:00:27.180150 15439 ts_tablet_manager.cc:1397] T 1461cdeda5664afd9c4c2b45199179b6 P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.157s user 0.010s sys 0.000s
I20260810 08:00:27.182680 15439 raft_consensus.cc:348] T 1461cdeda5664afd9c4c2b45199179b6 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:27.186165 15439 raft_consensus.cc:374] T 1461cdeda5664afd9c4c2b45199179b6 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:27.186324 15490 tablet_bootstrap.cc:492] T 728f8f80b3bb4b10871d8b400f25377b P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:27.189700 15439 raft_consensus.cc:729] T 1461cdeda5664afd9c4c2b45199179b6 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:27.190416 15439 consensus_queue.cc:260] T 1461cdeda5664afd9c4c2b45199179b6 P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:27.204600 15439 ts_tablet_manager.cc:1428] T 1461cdeda5664afd9c4c2b45199179b6 P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.024s user 0.005s sys 0.000s
I20260810 08:00:27.204825 14690 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_dfe235d424c64948a6330b4935504487" } new_table_name: "default.Table_beb88f8abaa447bfb6962df5718b7313" modify_external_catalogs: true
I20260810 08:00:27.208060 15439 tablet_bootstrap.cc:492] T 9ddae63b29c34e1397c5765e52e1478f P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:27.205528 15478 ts_tablet_manager.cc:1918] T abdd2b119a4b448dacdd2d87da34832f P 3fc5e568f3724f158b616c303efb9eae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:27.214221 15478 log.cc:1196] T abdd2b119a4b448dacdd2d87da34832f P 3fc5e568f3724f158b616c303efb9eae: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal/wals/abdd2b119a4b448dacdd2d87da34832f
I20260810 08:00:27.214854 15490 tablet_bootstrap.cc:654] T 728f8f80b3bb4b10871d8b400f25377b P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:27.216007 15478 ts_tablet_manager.cc:1939] T abdd2b119a4b448dacdd2d87da34832f P 3fc5e568f3724f158b616c303efb9eae: Deleting consensus metadata
I20260810 08:00:27.220635 15439 tablet_bootstrap.cc:654] T 9ddae63b29c34e1397c5765e52e1478f P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:27.221570 15482 ts_tablet_manager.cc:1918] T 472fbc9204754317aab059a4844d7de1 P a91b27762182454baada71e91ceab224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 08:00:27.225020 14678 catalog_manager.cc:4841] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): tablet abdd2b119a4b448dacdd2d87da34832f (table default.Table_3a39902b550c4390a56429d0ef94860e [id=cee29bc23d044a4bae553990b0d3c32c]) successfully deleted
W20260810 08:00:27.226559 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet 35b6da60a5544d7e9b657a5f1e6f90ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35b6da60a5544d7e9b657a5f1e6f90ff already in progress: creating tablet
I20260810 08:00:27.227329 15478 tablet_replica.cc:330] T 0c152aab7e244afd857534869b13815c P 3fc5e568f3724f158b616c303efb9eae: stopping tablet replica
I20260810 08:00:27.226183 15482 log.cc:1196] T 472fbc9204754317aab059a4844d7de1 P a91b27762182454baada71e91ceab224: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal/wals/472fbc9204754317aab059a4844d7de1
I20260810 08:00:27.228051 15478 raft_consensus.cc:2229] T 0c152aab7e244afd857534869b13815c P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:27.228526 15478 raft_consensus.cc:2258] T 0c152aab7e244afd857534869b13815c P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:27.230857 15482 ts_tablet_manager.cc:1939] T 472fbc9204754317aab059a4844d7de1 P a91b27762182454baada71e91ceab224: Deleting consensus metadata
I20260810 08:00:27.232759 15478 ts_tablet_manager.cc:1905] T 0c152aab7e244afd857534869b13815c P 3fc5e568f3724f158b616c303efb9eae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:27.234716 15490 tablet_bootstrap.cc:492] T 728f8f80b3bb4b10871d8b400f25377b P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:27.235373 15490 ts_tablet_manager.cc:1397] T 728f8f80b3bb4b10871d8b400f25377b P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.049s user 0.009s sys 0.002s
I20260810 08:00:27.236454 15604 raft_consensus.cc:482] T 114d3eac994f466bb1787e066de05e2c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:27.236886 15604 raft_consensus.cc:504] T 114d3eac994f466bb1787e066de05e2c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:27.238891 15604 leader_election.cc:288] T 114d3eac994f466bb1787e066de05e2c P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:27.239853 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "114d3eac994f466bb1787e066de05e2c" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:27.240357 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "114d3eac994f466bb1787e066de05e2c" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
W20260810 08:00:27.241446 15079 leader_election.cc:341] T 114d3eac994f466bb1787e066de05e2c P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
W20260810 08:00:27.242177 15076 leader_election.cc:341] T 114d3eac994f466bb1787e066de05e2c P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:27.242724 15076 leader_election.cc:302] T 114d3eac994f466bb1787e066de05e2c P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:27.243628 15604 raft_consensus.cc:2726] T 114d3eac994f466bb1787e066de05e2c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:27.248935 15437 tablet_bootstrap.cc:492] T a1c4f8f514c94128aedd07bd5efdc3be P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:27.249382 15437 ts_tablet_manager.cc:1397] T a1c4f8f514c94128aedd07bd5efdc3be P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.124s user 0.017s sys 0.001s
I20260810 08:00:27.249882 15482 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34630 (request call id 44) took 1041 ms. Trace:
I20260810 08:00:27.251024 15490 raft_consensus.cc:348] T 728f8f80b3bb4b10871d8b400f25377b P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:27.252239 15437 raft_consensus.cc:348] T a1c4f8f514c94128aedd07bd5efdc3be P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:27.252902 15437 raft_consensus.cc:374] T a1c4f8f514c94128aedd07bd5efdc3be P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:27.252985 15490 raft_consensus.cc:374] T 728f8f80b3bb4b10871d8b400f25377b P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:27.253140 15437 raft_consensus.cc:729] T a1c4f8f514c94128aedd07bd5efdc3be P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:27.251751 15482 rpcz_store.cc:276] 0810 08:00:26.208669 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:26.208818 (+ 149us) service_pool.cc:227] Handling call
0810 08:00:27.028092 (+819274us) ts_tablet_manager.cc:1145] Deleting tablet 472fbc9204754317aab059a4844d7de1
0810 08:00:27.028136 (+ 44us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:27.190593 (+162457us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:27.221339 (+ 30746us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:27.249817 (+ 28478us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":818725}
I20260810 08:00:27.254218 15490 raft_consensus.cc:729] T 728f8f80b3bb4b10871d8b400f25377b P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:27.256979 14681 catalog_manager.cc:4841] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): tablet 472fbc9204754317aab059a4844d7de1 (table default.Table_eab0fdadaf49486ca724a0a6e179a889 [id=fc52022e20044cc3bc88ac143e088e69]) successfully deleted
I20260810 08:00:27.257261 14694 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_1782c242ad164aa69cb679bbb072934e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:27.258195 15490 consensus_queue.cc:260] T 728f8f80b3bb4b10871d8b400f25377b P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
W20260810 08:00:27.259023 14694 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1782c242ad164aa69cb679bbb072934e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:27.255553 15479 ts_tablet_manager.cc:1918] T 2faab63c5023431c98e7b01ce61e04c2 P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:27.261446 15479 log.cc:1196] T 2faab63c5023431c98e7b01ce61e04c2 P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/2faab63c5023431c98e7b01ce61e04c2
I20260810 08:00:27.261309 15437 consensus_queue.cc:260] T a1c4f8f514c94128aedd07bd5efdc3be P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:27.262235 15479 ts_tablet_manager.cc:1939] T 2faab63c5023431c98e7b01ce61e04c2 P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:27.260689 15482 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34630 (request call id 45) took 1048 ms. Trace:
I20260810 08:00:27.270691 15439 tablet_bootstrap.cc:492] T 9ddae63b29c34e1397c5765e52e1478f P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:27.271126 15439 ts_tablet_manager.cc:1397] T 9ddae63b29c34e1397c5765e52e1478f P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.063s user 0.011s sys 0.000s
I20260810 08:00:27.273686 15437 ts_tablet_manager.cc:1428] T a1c4f8f514c94128aedd07bd5efdc3be P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.024s user 0.006s sys 0.000s
I20260810 08:00:27.275131 15439 raft_consensus.cc:348] T 9ddae63b29c34e1397c5765e52e1478f P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:27.275779 15439 raft_consensus.cc:374] T 9ddae63b29c34e1397c5765e52e1478f P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:27.276017 15439 raft_consensus.cc:729] T 9ddae63b29c34e1397c5765e52e1478f P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:27.277384 15437 tablet_bootstrap.cc:492] T 114d3eac994f466bb1787e066de05e2c P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:27.265065 15482 rpcz_store.cc:276] 0810 08:00:26.212410 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:26.212552 (+ 142us) service_pool.cc:227] Handling call
0810 08:00:27.260167 (+1047615us) ts_tablet_manager.cc:1145] Deleting tablet 37d8a37a16f34a0285c43c0d71b5f072
0810 08:00:27.260236 (+ 69us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:27.260609 (+ 373us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1047040}
I20260810 08:00:27.278754 15482 tablet_replica.cc:330] T a246b8cad99b47a9a856f6aed70bdfc8 P a91b27762182454baada71e91ceab224: stopping tablet replica
W20260810 08:00:27.278658 14681 catalog_manager.cc:4821] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet 37d8a37a16f34a0285c43c0d71b5f072 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37d8a37a16f34a0285c43c0d71b5f072 already in progress: creating tablet
I20260810 08:00:27.278733 15439 consensus_queue.cc:260] T 9ddae63b29c34e1397c5765e52e1478f P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:27.279397 15482 raft_consensus.cc:2229] T a246b8cad99b47a9a856f6aed70bdfc8 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:27.279786 15482 raft_consensus.cc:2258] T a246b8cad99b47a9a856f6aed70bdfc8 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:27.279807 14688 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_b78f4082645e4c51ad4d4b487939ffee" } new_table_name: "default.Table_1567aaa1d2ab44028745e82c29232682" modify_external_catalogs: true
I20260810 08:00:27.281199 15482 ts_tablet_manager.cc:1905] T a246b8cad99b47a9a856f6aed70bdfc8 P a91b27762182454baada71e91ceab224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:27.263971 15479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43002 (request call id 32) took 2056 ms. Trace:
I20260810 08:00:27.276820 15490 ts_tablet_manager.cc:1428] T 728f8f80b3bb4b10871d8b400f25377b P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.041s user 0.002s sys 0.005s
I20260810 08:00:27.290225 15439 ts_tablet_manager.cc:1428] T 9ddae63b29c34e1397c5765e52e1478f P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20260810 08:00:27.295450 15439 tablet_bootstrap.cc:492] T 2faab63c5023431c98e7b01ce61e04c2 P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:27.299234 15490 tablet_bootstrap.cc:492] T 0fae4809ef5e4a9fb16976674e287f03 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:27.282516 15479 rpcz_store.cc:276] 0810 08:00:25.207975 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:25.208136 (+ 161us) service_pool.cc:227] Handling call
0810 08:00:27.136733 (+1928597us) ts_tablet_manager.cc:1145] Deleting tablet 2faab63c5023431c98e7b01ce61e04c2
0810 08:00:27.136773 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:27.201965 (+ 65192us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:27.255300 (+ 53335us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:27.263918 (+ 8618us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1927941}
I20260810 08:00:27.304423 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet 2faab63c5023431c98e7b01ce61e04c2 (table default.Table_10462271d83b4063b56a4cd0c8b6f94a [id=67e4dae7d49b48669b176960841b25ab]) successfully deleted
I20260810 08:00:27.304790 15437 tablet_bootstrap.cc:654] T 114d3eac994f466bb1787e066de05e2c P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:27.305351 15479 tablet_replica.cc:330] T 114d3eac994f466bb1787e066de05e2c P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:27.306236 15479 raft_consensus.cc:2229] T 114d3eac994f466bb1787e066de05e2c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:27.306761 15479 raft_consensus.cc:2258] T 114d3eac994f466bb1787e066de05e2c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:27.317955 15439 tablet_bootstrap.cc:654] T 2faab63c5023431c98e7b01ce61e04c2 P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:27.315955 15478 ts_tablet_manager.cc:1918] T 0c152aab7e244afd857534869b13815c P 3fc5e568f3724f158b616c303efb9eae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 08:00:27.320636 15478 log.cc:1196] T 0c152aab7e244afd857534869b13815c P 3fc5e568f3724f158b616c303efb9eae: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal/wals/0c152aab7e244afd857534869b13815c
I20260810 08:00:27.321113 15479 ts_tablet_manager.cc:1905] T 114d3eac994f466bb1787e066de05e2c P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:27.321341 15478 ts_tablet_manager.cc:1939] T 0c152aab7e244afd857534869b13815c P 3fc5e568f3724f158b616c303efb9eae: Deleting consensus metadata
I20260810 08:00:27.324317 15478 tablet_replica.cc:330] T a246b8cad99b47a9a856f6aed70bdfc8 P 3fc5e568f3724f158b616c303efb9eae: stopping tablet replica
I20260810 08:00:27.324164 14678 catalog_manager.cc:4841] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): tablet 0c152aab7e244afd857534869b13815c (table default.Table_8ac532f4969e45a197db311de2a7b795 [id=7893824f2c374508b7b3c4635ebea5b6]) successfully deleted
I20260810 08:00:27.324954 15478 raft_consensus.cc:2229] T a246b8cad99b47a9a856f6aed70bdfc8 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:27.325366 15478 raft_consensus.cc:2258] T a246b8cad99b47a9a856f6aed70bdfc8 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:27.326373 15490 tablet_bootstrap.cc:654] T 0fae4809ef5e4a9fb16976674e287f03 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:27.327001 15478 ts_tablet_manager.cc:1905] T a246b8cad99b47a9a856f6aed70bdfc8 P 3fc5e568f3724f158b616c303efb9eae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:27.326845 15278 tablet_service.cc:1467] Processing CreateTablet for tablet d0761b090d8147b8a5537b0ed51108a9 (DEFAULT_TABLE table=default.Table_f387e847280a4f048905e15c5003f58c [id=fc7978f30c9a49c8aef649a6d10d87ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:27.327931 15278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0761b090d8147b8a5537b0ed51108a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:27.327004 14692 catalog_manager.cc:5495] T 1461cdeda5664afd9c4c2b45199179b6 P a91b27762182454baada71e91ceab224 reported cstate change: term changed from 0 to 1, leader changed from <none> to a91b27762182454baada71e91ceab224 (127.14.57.193). New cstate: current_term: 1 leader_uuid: "a91b27762182454baada71e91ceab224" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } health_report { overall_health: UNKNOWN } } }
I20260810 08:00:27.334740 15437 tablet_bootstrap.cc:492] T 114d3eac994f466bb1787e066de05e2c P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:27.335150 15437 ts_tablet_manager.cc:1397] T 114d3eac994f466bb1787e066de05e2c P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.058s user 0.010s sys 0.000s
I20260810 08:00:27.337440 15437 raft_consensus.cc:348] T 114d3eac994f466bb1787e066de05e2c P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:27.338114 15437 raft_consensus.cc:374] T 114d3eac994f466bb1787e066de05e2c P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:27.338354 15437 raft_consensus.cc:729] T 114d3eac994f466bb1787e066de05e2c P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:27.339107 15437 consensus_queue.cc:260] T 114d3eac994f466bb1787e066de05e2c P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
W20260810 08:00:27.341491 15076 consensus_peers.cc:487] T 0c152aab7e244afd857534869b13815c P 9d7219686f304f62a54acc7497cbbd07 -> Peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Couldn't send request to peer 3fc5e568f3724f158b616c303efb9eae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0c152aab7e244afd857534869b13815c. This is attempt 1: this message will repeat every 5th retry.
I20260810 08:00:27.345469 15122 tablet_service.cc:1467] Processing CreateTablet for tablet d0761b090d8147b8a5537b0ed51108a9 (DEFAULT_TABLE table=default.Table_f387e847280a4f048905e15c5003f58c [id=fc7978f30c9a49c8aef649a6d10d87ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:27.347060 15122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0761b090d8147b8a5537b0ed51108a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:27.350419 15490 tablet_bootstrap.cc:492] T 0fae4809ef5e4a9fb16976674e287f03 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:27.350826 15490 ts_tablet_manager.cc:1397] T 0fae4809ef5e4a9fb16976674e287f03 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.052s user 0.009s sys 0.001s
I20260810 08:00:27.352523 14965 tablet_service.cc:1467] Processing CreateTablet for tablet d0761b090d8147b8a5537b0ed51108a9 (DEFAULT_TABLE table=default.Table_f387e847280a4f048905e15c5003f58c [id=fc7978f30c9a49c8aef649a6d10d87ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:27.353649 14965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0761b090d8147b8a5537b0ed51108a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:27.354621 15437 ts_tablet_manager.cc:1428] T 114d3eac994f466bb1787e066de05e2c P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20260810 08:00:27.356928 15437 tablet_bootstrap.cc:492] T 5a22a48e2ba843b3a32a8900ab937ea0 P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:27.359066 15490 raft_consensus.cc:348] T 0fae4809ef5e4a9fb16976674e287f03 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:27.359720 15490 raft_consensus.cc:374] T 0fae4809ef5e4a9fb16976674e287f03 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:27.359961 15490 raft_consensus.cc:729] T 0fae4809ef5e4a9fb16976674e287f03 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:27.360497 15490 consensus_queue.cc:260] T 0fae4809ef5e4a9fb16976674e287f03 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:27.364794 15437 tablet_bootstrap.cc:654] T 5a22a48e2ba843b3a32a8900ab937ea0 P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:27.368438 15278 tablet_service.cc:1467] Processing CreateTablet for tablet 2d295305b5a843eaa84c8ad666355b84 (DEFAULT_TABLE table=default.Table_d5c08115c8b24ad48a9686e48a45ab82 [id=bb6d5450013a45b1bb6a85bc9e4e6d9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:27.371332 14962 tablet_service.cc:1467] Processing CreateTablet for tablet 2d295305b5a843eaa84c8ad666355b84 (DEFAULT_TABLE table=default.Table_d5c08115c8b24ad48a9686e48a45ab82 [id=bb6d5450013a45b1bb6a85bc9e4e6d9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:27.371330 15275 tablet_service.cc:1467] Processing CreateTablet for tablet 25de7bf64bf6450ba17b7748568bc0ed (DEFAULT_TABLE table=default.Table_1732e711b4dd4f3392ee652299146573 [id=d1b667b58f924e71adecc1f47806bee5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:27.371942 15490 ts_tablet_manager.cc:1428] T 0fae4809ef5e4a9fb16976674e287f03 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.021s user 0.005s sys 0.001s
I20260810 08:00:27.372429 14962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d295305b5a843eaa84c8ad666355b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:27.372161 15278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d295305b5a843eaa84c8ad666355b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:27.372879 15490 tablet_bootstrap.cc:492] T 37d8a37a16f34a0285c43c0d71b5f072 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:27.373597 15275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25de7bf64bf6450ba17b7748568bc0ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:27.377074 15447 raft_consensus.cc:482] T 9ddae63b29c34e1397c5765e52e1478f P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:27.377496 15447 raft_consensus.cc:504] T 9ddae63b29c34e1397c5765e52e1478f P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:27.378196 15482 ts_tablet_manager.cc:1918] T a246b8cad99b47a9a856f6aed70bdfc8 P a91b27762182454baada71e91ceab224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:27.378541 15482 log.cc:1196] T a246b8cad99b47a9a856f6aed70bdfc8 P a91b27762182454baada71e91ceab224: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal/wals/a246b8cad99b47a9a856f6aed70bdfc8
I20260810 08:00:27.378407 14963 tablet_service.cc:1467] Processing CreateTablet for tablet 25de7bf64bf6450ba17b7748568bc0ed (DEFAULT_TABLE table=default.Table_1732e711b4dd4f3392ee652299146573 [id=d1b667b58f924e71adecc1f47806bee5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:27.379522 14963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25de7bf64bf6450ba17b7748568bc0ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:27.379510 15447 leader_election.cc:288] T 9ddae63b29c34e1397c5765e52e1478f P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
W20260810 08:00:27.382364 15234 leader_election.cc:341] T 9ddae63b29c34e1397c5765e52e1478f P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Not found: Tablet not found: 9ddae63b29c34e1397c5765e52e1478f
I20260810 08:00:27.384560 15482 ts_tablet_manager.cc:1939] T a246b8cad99b47a9a856f6aed70bdfc8 P a91b27762182454baada71e91ceab224: Deleting consensus metadata
I20260810 08:00:27.386970 15482 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34630 (request call id 47) took 1052 ms. Trace:
I20260810 08:00:27.387279 15482 rpcz_store.cc:276] 0810 08:00:26.334298 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:26.338859 (+ 4561us) service_pool.cc:227] Handling call
0810 08:00:27.278611 (+939752us) ts_tablet_manager.cc:1145] Deleting tablet a246b8cad99b47a9a856f6aed70bdfc8
0810 08:00:27.278652 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:27.342464 (+ 63812us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:27.377979 (+ 35515us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:27.386909 (+ 8930us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":939202}
I20260810 08:00:27.388515 14681 catalog_manager.cc:4841] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): tablet a246b8cad99b47a9a856f6aed70bdfc8 (table default.Table_491d9caa5f6d4e3cb934a5d0582b519c [id=d106e75991f54f60abbe60b0495cb8e8]) successfully deleted
I20260810 08:00:27.391459 15482 tablet_replica.cc:330] T 35b6da60a5544d7e9b657a5f1e6f90ff P a91b27762182454baada71e91ceab224: stopping tablet replica
I20260810 08:00:27.394737 15274 tablet_service.cc:1467] Processing CreateTablet for tablet d22a778bfe784b2f8a7b4d2216909f10 (DEFAULT_TABLE table=default.Table_b7ee9352799b44f4b7bbf48355b3af11 [id=f008513c7e134e1992c6c8ad3e757bbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:27.396203 15274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d22a778bfe784b2f8a7b4d2216909f10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:27.399294 14966 tablet_service.cc:1467] Processing CreateTablet for tablet d22a778bfe784b2f8a7b4d2216909f10 (DEFAULT_TABLE table=default.Table_b7ee9352799b44f4b7bbf48355b3af11 [id=f008513c7e134e1992c6c8ad3e757bbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:27.400466 14966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d22a778bfe784b2f8a7b4d2216909f10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:27.402647 14964 tablet_service.cc:1467] Processing CreateTablet for tablet 2cca8a18778c49d0be01363c31ef180b (DEFAULT_TABLE table=default.Table_1e20f010e84746b0a487c82539f9ab8a [id=82b31a19f2724a95938f0ae5b4700d97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:27.406452 15279 tablet_service.cc:1467] Processing CreateTablet for tablet 2cca8a18778c49d0be01363c31ef180b (DEFAULT_TABLE table=default.Table_1e20f010e84746b0a487c82539f9ab8a [id=82b31a19f2724a95938f0ae5b4700d97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:27.407658 15279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cca8a18778c49d0be01363c31ef180b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:27.407477 14691 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_3d528c5b4aa54991bf128ada67924546"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:27.409735 14691 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d528c5b4aa54991bf128ada67924546 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:27.416575 15482 raft_consensus.cc:2229] T 35b6da60a5544d7e9b657a5f1e6f90ff P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:27.417208 15482 raft_consensus.cc:2258] T 35b6da60a5544d7e9b657a5f1e6f90ff P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:27.417315 14964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cca8a18778c49d0be01363c31ef180b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:27.425642 15482 ts_tablet_manager.cc:1905] T 35b6da60a5544d7e9b657a5f1e6f90ff P a91b27762182454baada71e91ceab224: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 08:00:27.436386 14690 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0fae4809ef5e4a9fb16976674e287f03 consensus state has no leader
I20260810 08:00:27.450572 15121 tablet_service.cc:1467] Processing CreateTablet for tablet 2cca8a18778c49d0be01363c31ef180b (DEFAULT_TABLE table=default.Table_1e20f010e84746b0a487c82539f9ab8a [id=82b31a19f2724a95938f0ae5b4700d97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:27.452284 15121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cca8a18778c49d0be01363c31ef180b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:27.462754 15123 tablet_service.cc:1467] Processing CreateTablet for tablet 25de7bf64bf6450ba17b7748568bc0ed (DEFAULT_TABLE table=default.Table_1732e711b4dd4f3392ee652299146573 [id=d1b667b58f924e71adecc1f47806bee5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:27.464148 15123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25de7bf64bf6450ba17b7748568bc0ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:27.464291 15147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ddae63b29c34e1397c5765e52e1478f" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:27.467150 15126 tablet_service.cc:1467] Processing CreateTablet for tablet 2d295305b5a843eaa84c8ad666355b84 (DEFAULT_TABLE table=default.Table_d5c08115c8b24ad48a9686e48a45ab82 [id=bb6d5450013a45b1bb6a85bc9e4e6d9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:27.468444 15126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d295305b5a843eaa84c8ad666355b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:27.470643 15234 leader_election.cc:302] T 9ddae63b29c34e1397c5765e52e1478f P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae; no voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224
I20260810 08:00:27.471858 15447 raft_consensus.cc:2726] T 9ddae63b29c34e1397c5765e52e1478f P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:27.475659 15125 tablet_service.cc:1467] Processing CreateTablet for tablet d22a778bfe784b2f8a7b4d2216909f10 (DEFAULT_TABLE table=default.Table_b7ee9352799b44f4b7bbf48355b3af11 [id=f008513c7e134e1992c6c8ad3e757bbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:27.475600 14690 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_5e2233b19d3d486a86f8dc7190f9e2f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:27.476917 15125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d22a778bfe784b2f8a7b4d2216909f10. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 08:00:27.477450 14690 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e2233b19d3d486a86f8dc7190f9e2f1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:27.486914 15490 tablet_bootstrap.cc:700] T 37d8a37a16f34a0285c43c0d71b5f072 P 9d7219686f304f62a54acc7497cbbd07: Time spent opening tablet: real 0.110s user 0.004s sys 0.000s
I20260810 08:00:27.487211 15573 raft_consensus.cc:482] T 5a22a48e2ba843b3a32a8900ab937ea0 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:27.487924 15490 tablet_bootstrap.cc:654] T 37d8a37a16f34a0285c43c0d71b5f072 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:27.488312 15437 tablet_bootstrap.cc:492] T 5a22a48e2ba843b3a32a8900ab937ea0 P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:27.487812 15573 raft_consensus.cc:504] T 5a22a48e2ba843b3a32a8900ab937ea0 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:27.488726 15437 ts_tablet_manager.cc:1397] T 5a22a48e2ba843b3a32a8900ab937ea0 P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.132s user 0.008s sys 0.000s
I20260810 08:00:27.490923 15437 raft_consensus.cc:348] T 5a22a48e2ba843b3a32a8900ab937ea0 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:27.491561 15437 raft_consensus.cc:374] T 5a22a48e2ba843b3a32a8900ab937ea0 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:27.491806 15437 raft_consensus.cc:729] T 5a22a48e2ba843b3a32a8900ab937ea0 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:27.494226 15437 consensus_queue.cc:260] T 5a22a48e2ba843b3a32a8900ab937ea0 P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:27.505204 15276 tablet_service.cc:1467] Processing CreateTablet for tablet 68bc39707f44457fbf57c217ad827348 (DEFAULT_TABLE table=default.Table_28276fa7c9a042209ceaf8cc7c13eb43 [id=fbd03ee96b004797a503841f4cc7249a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:27.506844 15276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68bc39707f44457fbf57c217ad827348. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:27.507508 15275 tablet_service.cc:1467] Processing CreateTablet for tablet cf8bb1c44a7b4d17bddeff8b2fe81b35 (DEFAULT_TABLE table=default.Table_a6ce85a21a354e7c92f158c1ea3212b0 [id=8329ae05a9b0470897f5830376c1f87a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:27.508553 15275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf8bb1c44a7b4d17bddeff8b2fe81b35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:27.512603 14964 tablet_service.cc:1467] Processing CreateTablet for tablet cf8bb1c44a7b4d17bddeff8b2fe81b35 (DEFAULT_TABLE table=default.Table_a6ce85a21a354e7c92f158c1ea3212b0 [id=8329ae05a9b0470897f5830376c1f87a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:27.513901 14964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf8bb1c44a7b4d17bddeff8b2fe81b35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:27.503767 14966 tablet_service.cc:1467] Processing CreateTablet for tablet 4381ccb8711744ff8f9766ef49c53e28 (DEFAULT_TABLE table=default.Table_5e1d024b54504287840e8c45c056fa56 [id=f07affbf494748a88874b8f1edd3f7b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:27.517108 14966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4381ccb8711744ff8f9766ef49c53e28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:27.515594 15278 tablet_service.cc:1467] Processing CreateTablet for tablet 4381ccb8711744ff8f9766ef49c53e28 (DEFAULT_TABLE table=default.Table_5e1d024b54504287840e8c45c056fa56 [id=f07affbf494748a88874b8f1edd3f7b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:27.518548 15278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4381ccb8711744ff8f9766ef49c53e28. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 08:00:27.520141 14681 consensus_peers.cc:487] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 -> Peer 1e8cadb26fe0428a964f66ef8172f3be (127.14.57.252:33819): Couldn't send request to peer 1e8cadb26fe0428a964f66ef8172f3be. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.252:33819: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260810 08:00:27.547210 15079 consensus_peers.cc:487] T 9ddae63b29c34e1397c5765e52e1478f P 9d7219686f304f62a54acc7497cbbd07 -> Peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Couldn't send request to peer a91b27762182454baada71e91ceab224. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9ddae63b29c34e1397c5765e52e1478f. This is attempt 1: this message will repeat every 5th retry.
I20260810 08:00:27.549903 15479 ts_tablet_manager.cc:1918] T 114d3eac994f466bb1787e066de05e2c P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:27.550478 15479 log.cc:1196] T 114d3eac994f466bb1787e066de05e2c P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/114d3eac994f466bb1787e066de05e2c
I20260810 08:00:27.551435 15479 ts_tablet_manager.cc:1939] T 114d3eac994f466bb1787e066de05e2c P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:27.552462 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a22a48e2ba843b3a32a8900ab937ea0" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:27.552557 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a22a48e2ba843b3a32a8900ab937ea0" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:27.553500 14987 raft_consensus.cc:2454] T 5a22a48e2ba843b3a32a8900ab937ea0 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d7219686f304f62a54acc7497cbbd07 in term 0.
I20260810 08:00:27.555094 15120 tablet_service.cc:1467] Processing CreateTablet for tablet 4381ccb8711744ff8f9766ef49c53e28 (DEFAULT_TABLE table=default.Table_5e1d024b54504287840e8c45c056fa56 [id=f07affbf494748a88874b8f1edd3f7b6]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 08:00:27.556581 15076 leader_election.cc:341] T 5a22a48e2ba843b3a32a8900ab937ea0 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:27.560524 15479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43002 (request call id 36) took 1877 ms. Trace:
I20260810 08:00:27.562727 15274 tablet_service.cc:1467] Processing CreateTablet for tablet 924c0f73638f48a88c08d45d459132e1 (DEFAULT_TABLE table=default.Table_409addd050ee45b981bddfacfab24c03 [id=76166bd7cb584067b7d2e305603a840b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:27.564970 15573 leader_election.cc:288] T 5a22a48e2ba843b3a32a8900ab937ea0 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:27.565336 14963 tablet_service.cc:1467] Processing CreateTablet for tablet b5a0e0af742340b2945be2102768b307 (DEFAULT_TABLE table=default.Table_eab0fdadaf49486ca724a0a6e179a889 [id=fc52022e20044cc3bc88ac143e088e69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:27.565357 15120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4381ccb8711744ff8f9766ef49c53e28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:27.567082 14963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5a0e0af742340b2945be2102768b307. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:27.578456 14962 tablet_service.cc:1467] Processing CreateTablet for tablet 68bc39707f44457fbf57c217ad827348 (DEFAULT_TABLE table=default.Table_28276fa7c9a042209ceaf8cc7c13eb43 [id=fbd03ee96b004797a503841f4cc7249a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:27.579706 14962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68bc39707f44457fbf57c217ad827348. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:27.579156 15119 tablet_service.cc:1467] Processing CreateTablet for tablet cf8bb1c44a7b4d17bddeff8b2fe81b35 (DEFAULT_TABLE table=default.Table_a6ce85a21a354e7c92f158c1ea3212b0 [id=8329ae05a9b0470897f5830376c1f87a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:27.580951 15119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf8bb1c44a7b4d17bddeff8b2fe81b35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:27.583263 15274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 924c0f73638f48a88c08d45d459132e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:27.561210 15479 rpcz_store.cc:276] 0810 08:00:25.682844 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:25.683009 (+ 165us) service_pool.cc:227] Handling call
0810 08:00:27.305195 (+1622186us) ts_tablet_manager.cc:1145] Deleting tablet 114d3eac994f466bb1787e066de05e2c
0810 08:00:27.305240 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:27.336255 (+ 31015us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:27.486353 (+150098us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:27.560420 (+ 74067us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1621560}
I20260810 08:00:27.586427 14965 tablet_service.cc:1467] Processing CreateTablet for tablet 924c0f73638f48a88c08d45d459132e1 (DEFAULT_TABLE table=default.Table_409addd050ee45b981bddfacfab24c03 [id=76166bd7cb584067b7d2e305603a840b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:27.587862 14965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 924c0f73638f48a88c08d45d459132e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:27.591965 15118 tablet_service.cc:1467] Processing CreateTablet for tablet 68bc39707f44457fbf57c217ad827348 (DEFAULT_TABLE table=default.Table_28276fa7c9a042209ceaf8cc7c13eb43 [id=fbd03ee96b004797a503841f4cc7249a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:27.596403 15117 tablet_service.cc:1467] Processing CreateTablet for tablet 924c0f73638f48a88c08d45d459132e1 (DEFAULT_TABLE table=default.Table_409addd050ee45b981bddfacfab24c03 [id=76166bd7cb584067b7d2e305603a840b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:27.598057 15117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 924c0f73638f48a88c08d45d459132e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:27.600131 15118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68bc39707f44457fbf57c217ad827348. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:27.602495 15482 ts_tablet_manager.cc:1918] T 35b6da60a5544d7e9b657a5f1e6f90ff P a91b27762182454baada71e91ceab224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:27.600723 15079 leader_election.cc:302] T 5a22a48e2ba843b3a32a8900ab937ea0 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224; no voters: 3fc5e568f3724f158b616c303efb9eae
I20260810 08:00:27.625782 14695 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet 2cca8a18778c49d0be01363c31ef180b
I20260810 08:00:27.630949 15279 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cca8a18778c49d0be01363c31ef180b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:26 UTC) from {username='slave'} at 127.0.0.1:40692
W20260810 08:00:27.631368 14688 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 49c35b7f4a72437fa0675abe8a8f3a71 consensus state has no leader
I20260810 08:00:27.634267 14693 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet 728f8f80b3bb4b10871d8b400f25377b
I20260810 08:00:27.635658 14693 catalog_manager.cc:6565] ReplaceTablet: tablet 728f8f80b3bb4b10871d8b400f25377b deleted and replaced by tablet e13b91d61063446f9d1454da4f8bb706
I20260810 08:00:27.635882 15439 tablet_bootstrap.cc:492] T 2faab63c5023431c98e7b01ce61e04c2 P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:27.636264 15439 ts_tablet_manager.cc:1397] T 2faab63c5023431c98e7b01ce61e04c2 P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.341s user 0.001s sys 0.026s
I20260810 08:00:27.638505 15279 tablet_service.cc:1514] Processing DeleteTablet for tablet 728f8f80b3bb4b10871d8b400f25377b with delete_type TABLET_DATA_DELETED (replaced by tablet e13b91d61063446f9d1454da4f8bb706) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:27.640252 15439 raft_consensus.cc:348] T 2faab63c5023431c98e7b01ce61e04c2 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:27.640946 15439 raft_consensus.cc:374] T 2faab63c5023431c98e7b01ce61e04c2 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:27.641219 15439 raft_consensus.cc:729] T 2faab63c5023431c98e7b01ce61e04c2 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:27.641985 15439 consensus_queue.cc:260] T 2faab63c5023431c98e7b01ce61e04c2 P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:27.643560 15478 ts_tablet_manager.cc:1918] T a246b8cad99b47a9a856f6aed70bdfc8 P 3fc5e568f3724f158b616c303efb9eae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:27.643878 15478 log.cc:1196] T a246b8cad99b47a9a856f6aed70bdfc8 P 3fc5e568f3724f158b616c303efb9eae: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal/wals/a246b8cad99b47a9a856f6aed70bdfc8
I20260810 08:00:27.644047 15439 ts_tablet_manager.cc:1428] T 2faab63c5023431c98e7b01ce61e04c2 P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.006s user 0.003s sys 0.001s
I20260810 08:00:27.644555 15478 ts_tablet_manager.cc:1939] T a246b8cad99b47a9a856f6aed70bdfc8 P 3fc5e568f3724f158b616c303efb9eae: Deleting consensus metadata
I20260810 08:00:27.649020 14678 catalog_manager.cc:4841] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): tablet a246b8cad99b47a9a856f6aed70bdfc8 (table default.Table_491d9caa5f6d4e3cb934a5d0582b519c [id=d106e75991f54f60abbe60b0495cb8e8]) successfully deleted
I20260810 08:00:27.650409 15478 tablet_replica.cc:330] T 472fbc9204754317aab059a4844d7de1 P 3fc5e568f3724f158b616c303efb9eae: stopping tablet replica
I20260810 08:00:27.651508 15478 raft_consensus.cc:2229] T 472fbc9204754317aab059a4844d7de1 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:27.652803 15478 raft_consensus.cc:2258] T 472fbc9204754317aab059a4844d7de1 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:27.664863 15279 tablet_service.cc:1467] Processing CreateTablet for tablet b5a0e0af742340b2945be2102768b307 (DEFAULT_TABLE table=default.Table_eab0fdadaf49486ca724a0a6e179a889 [id=fc52022e20044cc3bc88ac143e088e69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:27.665421 15439 tablet_bootstrap.cc:492] T 35b6da60a5544d7e9b657a5f1e6f90ff P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:27.665943 15478 ts_tablet_manager.cc:1905] T 472fbc9204754317aab059a4844d7de1 P 3fc5e568f3724f158b616c303efb9eae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:27.666321 15279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5a0e0af742340b2945be2102768b307. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:27.517165 15437 ts_tablet_manager.cc:1428] T 5a22a48e2ba843b3a32a8900ab937ea0 P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20260810 08:00:27.632344 14967 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cca8a18778c49d0be01363c31ef180b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:26 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:27.640204 14961 tablet_service.cc:1514] Processing DeleteTablet for tablet 728f8f80b3bb4b10871d8b400f25377b with delete_type TABLET_DATA_DELETED (replaced by tablet e13b91d61063446f9d1454da4f8bb706) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:27.670238 14960 tablet_service.cc:1467] Processing CreateTablet for tablet 561cbc6a4eb14d3584f88ea26a0fea7a (DEFAULT_TABLE table=default.Table_0e30c621046242609c9f6fc73b16d0e7 [id=e9608d66a9964bc6aafdf84f574fba02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:27.673067 15437 tablet_bootstrap.cc:492] T 2faab63c5023431c98e7b01ce61e04c2 P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:27.675057 15482 log.cc:1196] T 35b6da60a5544d7e9b657a5f1e6f90ff P a91b27762182454baada71e91ceab224: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal/wals/35b6da60a5544d7e9b657a5f1e6f90ff
I20260810 08:00:27.675297 14960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 561cbc6a4eb14d3584f88ea26a0fea7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:27.675750 15482 ts_tablet_manager.cc:1939] T 35b6da60a5544d7e9b657a5f1e6f90ff P a91b27762182454baada71e91ceab224: Deleting consensus metadata
I20260810 08:00:27.695053 14693 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_3099152ccd3b4d5a8b558ee964aefaf8" } new_table_name: "default.Table_e6b03850596042a6a30ee45ea0720b86" modify_external_catalogs: true
I20260810 08:00:27.703534 15276 tablet_service.cc:1467] Processing CreateTablet for tablet 561cbc6a4eb14d3584f88ea26a0fea7a (DEFAULT_TABLE table=default.Table_0e30c621046242609c9f6fc73b16d0e7 [id=e9608d66a9964bc6aafdf84f574fba02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:27.703534 15277 tablet_service.cc:1467] Processing CreateTablet for tablet 1f427f4093e5417a9ca79b1805b0675e (DEFAULT_TABLE table=default.Table_f0a0a164132a4d46891036b0fee9839a [id=11ca1b81b7704d33820eef06f5a25310]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:27.700752 14695 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_69ee062d338749dbba588a79aee54e1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:27.704665 15439 tablet_bootstrap.cc:654] T 35b6da60a5544d7e9b657a5f1e6f90ff P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:27.704943 15277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f427f4093e5417a9ca79b1805b0675e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:27.704742 15280 tablet_service.cc:1467] Processing CreateTablet for tablet cb37a3a87f8b48a68c19510f306073c2 (DEFAULT_TABLE table=default.Table_64e1c230e9624fc9ba6c328346150ffe [id=46e6c2115d8c4b5e9907ed76b273638a]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 08:00:27.705744 14695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69ee062d338749dbba588a79aee54e1e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:27.705844 15280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb37a3a87f8b48a68c19510f306073c2. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 08:00:27.705971 14680 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 49c35b7f4a72437fa0675abe8a8f3a71 (table default.Table_1567aaa1d2ab44028745e82c29232682 [id=6cf63d3e931a4f6487074494254805bb], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet 49c35b7f4a72437fa0675abe8a8f3a71 consensus state has no leader
I20260810 08:00:27.715608 15490 tablet_bootstrap.cc:492] T 37d8a37a16f34a0285c43c0d71b5f072 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:27.716085 15490 ts_tablet_manager.cc:1397] T 37d8a37a16f34a0285c43c0d71b5f072 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.343s user 0.012s sys 0.017s
I20260810 08:00:27.722008 15479 tablet_replica.cc:330] T dc3ea02864d040c5bcde38b1a5609676 P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:27.723728 15276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 561cbc6a4eb14d3584f88ea26a0fea7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:27.717183 14961 tablet_service.cc:1467] Processing CreateTablet for tablet 1f427f4093e5417a9ca79b1805b0675e (DEFAULT_TABLE table=default.Table_f0a0a164132a4d46891036b0fee9839a [id=11ca1b81b7704d33820eef06f5a25310]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:27.724447 15482 tablet_replica.cc:330] T 0c152aab7e244afd857534869b13815c P a91b27762182454baada71e91ceab224: stopping tablet replica
I20260810 08:00:27.725121 15482 raft_consensus.cc:2229] T 0c152aab7e244afd857534869b13815c P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:27.725716 15482 raft_consensus.cc:2258] T 0c152aab7e244afd857534869b13815c P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:27.725227 15617 raft_consensus.cc:2781] T 5a22a48e2ba843b3a32a8900ab937ea0 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:27.726171 15617 raft_consensus.cc:482] T 5a22a48e2ba843b3a32a8900ab937ea0 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:27.726497 15617 raft_consensus.cc:3037] T 5a22a48e2ba843b3a32a8900ab937ea0 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:27.729866 14961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f427f4093e5417a9ca79b1805b0675e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:27.730713 14681 catalog_manager.cc:4841] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): tablet 35b6da60a5544d7e9b657a5f1e6f90ff (table default.Table_6e428ff2c4c7459b84d2882a089b490d [id=64f8570d723d4f5faba292689c535c01]) successfully deleted
W20260810 08:00:27.709899 15079 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.193:43601, user_credentials={real_user=slave}} blocked reactor thread for 104998us
I20260810 08:00:27.734040 15301 raft_consensus.cc:3037] T 1461cdeda5664afd9c4c2b45199179b6 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:27.750442 14960 tablet_service.cc:1467] Processing CreateTablet for tablet cb37a3a87f8b48a68c19510f306073c2 (DEFAULT_TABLE table=default.Table_64e1c230e9624fc9ba6c328346150ffe [id=46e6c2115d8c4b5e9907ed76b273638a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:27.750413 15443 consensus_queue.cc:1035] T 1461cdeda5664afd9c4c2b45199179b6 P a91b27762182454baada71e91ceab224 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 08:00:27.751639 14960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb37a3a87f8b48a68c19510f306073c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:27.759193 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet 114d3eac994f466bb1787e066de05e2c (table default.Table_6fcf472630534619ae18553e71ebedaf [id=2c633c1a27cf44f3ab04e66e2cba6e13]) successfully deleted
I20260810 08:00:27.766723 14694 catalog_manager.cc:5587] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265) does not have the latest schema for tablet 0fae4809ef5e4a9fb16976674e287f03 (table default.Table_beb88f8abaa447bfb6962df5718b7313 [id=41bcbdf2ef674d8b8db9284b4b0dcdb9]). Expected version 2 got 0
I20260810 08:00:27.767349 14694 catalog_manager.cc:5587] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265) does not have the latest schema for tablet 5a22a48e2ba843b3a32a8900ab937ea0 (table default.Table_5a6ab9b9acc9408c8811f77acaca9a0c [id=ac3a5787385940bca8030d7bac893a27]). Expected version 2 got 0
I20260810 08:00:27.767810 14694 catalog_manager.cc:5587] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265) does not have the latest schema for tablet a1c4f8f514c94128aedd07bd5efdc3be (table default.Table_89fe7ad723964551909b73ff6b72b9af [id=ce96e7fb7c46404a859ca08c1fd1b26a]). Expected version 1 got 0
I20260810 08:00:27.773290 15482 ts_tablet_manager.cc:1905] T 0c152aab7e244afd857534869b13815c P a91b27762182454baada71e91ceab224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:27.777937 15479 raft_consensus.cc:2229] T dc3ea02864d040c5bcde38b1a5609676 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:27.778493 15479 raft_consensus.cc:2258] T dc3ea02864d040c5bcde38b1a5609676 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:27.780386 15479 ts_tablet_manager.cc:1905] T dc3ea02864d040c5bcde38b1a5609676 P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:27.772890 15623 raft_consensus.cc:482] T a1c4f8f514c94128aedd07bd5efdc3be P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:27.781208 15623 raft_consensus.cc:504] T a1c4f8f514c94128aedd07bd5efdc3be P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:27.787983 15490 raft_consensus.cc:348] T 37d8a37a16f34a0285c43c0d71b5f072 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:27.816504 15623 leader_election.cc:288] T a1c4f8f514c94128aedd07bd5efdc3be P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:27.820405 15118 tablet_service.cc:1514] Processing DeleteTablet for tablet 728f8f80b3bb4b10871d8b400f25377b with delete_type TABLET_DATA_DELETED (replaced by tablet e13b91d61063446f9d1454da4f8bb706) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:27.823000 15118 tablet_service.cc:1467] Processing CreateTablet for tablet b5a0e0af742340b2945be2102768b307 (DEFAULT_TABLE table=default.Table_eab0fdadaf49486ca724a0a6e179a889 [id=fc52022e20044cc3bc88ac143e088e69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:27.824608 15118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5a0e0af742340b2945be2102768b307. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:27.827071 14687 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet d22a778bfe784b2f8a7b4d2216909f10
I20260810 08:00:27.832026 14686 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet fbf62f4eab8b49f282df3ab4d95d886f
I20260810 08:00:27.833429 14686 catalog_manager.cc:6565] ReplaceTablet: tablet fbf62f4eab8b49f282df3ab4d95d886f deleted and replaced by tablet e1b42eac370145d4b684456639968485
I20260810 08:00:27.840618 14686 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_4ff94124f5cb4710a0ad3ff1781efcd3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:27.845750 14686 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ff94124f5cb4710a0ad3ff1781efcd3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:27.847071 15117 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cca8a18778c49d0be01363c31ef180b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:26 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:27.850286 15125 tablet_service.cc:1467] Processing CreateTablet for tablet 1f427f4093e5417a9ca79b1805b0675e (DEFAULT_TABLE table=default.Table_f0a0a164132a4d46891036b0fee9839a [id=11ca1b81b7704d33820eef06f5a25310]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:27.851919 15125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f427f4093e5417a9ca79b1805b0675e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:27.861884 15604 raft_consensus.cc:482] T e46648198f5343f7a65079887e5df159 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:27.861824 14687 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_f387e847280a4f048905e15c5003f58c" } modify_external_catalogs: true
I20260810 08:00:27.862849 14687 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_f387e847280a4f048905e15c5003f58c" } modify_external_catalogs: true
I20260810 08:00:27.862624 15604 raft_consensus.cc:504] T e46648198f5343f7a65079887e5df159 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:27.867187 15117 tablet_service.cc:1467] Processing CreateTablet for tablet 561cbc6a4eb14d3584f88ea26a0fea7a (DEFAULT_TABLE table=default.Table_0e30c621046242609c9f6fc73b16d0e7 [id=e9608d66a9964bc6aafdf84f574fba02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:27.869794 15126 tablet_service.cc:1467] Processing CreateTablet for tablet cb37a3a87f8b48a68c19510f306073c2 (DEFAULT_TABLE table=default.Table_64e1c230e9624fc9ba6c328346150ffe [id=46e6c2115d8c4b5e9907ed76b273638a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:27.871411 15126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb37a3a87f8b48a68c19510f306073c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:27.872614 15437 tablet_bootstrap.cc:654] T 2faab63c5023431c98e7b01ce61e04c2 P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:27.878773 15122 tablet_service.cc:1514] Processing DeleteTablet for tablet d22a778bfe784b2f8a7b4d2216909f10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:26 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:27.836011 15617 raft_consensus.cc:504] T 5a22a48e2ba843b3a32a8900ab937ea0 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:27.881844 14964 tablet_service.cc:1514] Processing DeleteTablet for tablet d22a778bfe784b2f8a7b4d2216909f10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:26 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:27.798318 15490 raft_consensus.cc:374] T 37d8a37a16f34a0285c43c0d71b5f072 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:27.883571 15490 raft_consensus.cc:729] T 37d8a37a16f34a0285c43c0d71b5f072 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:27.885211 15623 raft_consensus.cc:482] T 728f8f80b3bb4b10871d8b400f25377b P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:27.885815 15623 raft_consensus.cc:504] T 728f8f80b3bb4b10871d8b400f25377b P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:27.893954 15623 leader_election.cc:288] T 728f8f80b3bb4b10871d8b400f25377b P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:27.894984 15122 tablet_service.cc:1514] Processing DeleteTablet for tablet fbf62f4eab8b49f282df3ab4d95d886f with delete_type TABLET_DATA_DELETED (replaced by tablet e1b42eac370145d4b684456639968485) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:27.905521 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1c4f8f514c94128aedd07bd5efdc3be" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:27.906565 14987 raft_consensus.cc:2454] T a1c4f8f514c94128aedd07bd5efdc3be P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d7219686f304f62a54acc7497cbbd07 in term 0.
I20260810 08:00:27.912187 14966 tablet_service.cc:1514] Processing DeleteTablet for tablet fbf62f4eab8b49f282df3ab4d95d886f with delete_type TABLET_DATA_DELETED (replaced by tablet e1b42eac370145d4b684456639968485) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:27.914738 15117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 561cbc6a4eb14d3584f88ea26a0fea7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:27.916424 15443 raft_consensus.cc:482] T 114d3eac994f466bb1787e066de05e2c P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:27.917049 15443 raft_consensus.cc:504] T 114d3eac994f466bb1787e066de05e2c P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:27.919687 15443 leader_election.cc:288] T 114d3eac994f466bb1787e066de05e2c P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:27.884629 15490 consensus_queue.cc:260] T 37d8a37a16f34a0285c43c0d71b5f072 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:27.924916 15490 ts_tablet_manager.cc:1428] T 37d8a37a16f34a0285c43c0d71b5f072 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.209s user 0.005s sys 0.003s
I20260810 08:00:27.926949 15478 ts_tablet_manager.cc:1918] T 472fbc9204754317aab059a4844d7de1 P 3fc5e568f3724f158b616c303efb9eae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:27.927364 15478 log.cc:1196] T 472fbc9204754317aab059a4844d7de1 P 3fc5e568f3724f158b616c303efb9eae: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal/wals/472fbc9204754317aab059a4844d7de1
I20260810 08:00:27.928020 15478 ts_tablet_manager.cc:1939] T 472fbc9204754317aab059a4844d7de1 P 3fc5e568f3724f158b616c303efb9eae: Deleting consensus metadata
I20260810 08:00:27.935072 15443 consensus_queue.cc:1035] T 1461cdeda5664afd9c4c2b45199179b6 P a91b27762182454baada71e91ceab224 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 08:00:27.937845 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e46648198f5343f7a65079887e5df159" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:27.934836 15300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "114d3eac994f466bb1787e066de05e2c" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:27.940177 15277 tablet_service.cc:1514] Processing DeleteTablet for tablet d22a778bfe784b2f8a7b4d2216909f10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:26 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:27.940804 15274 tablet_service.cc:1514] Processing DeleteTablet for tablet fbf62f4eab8b49f282df3ab4d95d886f with delete_type TABLET_DATA_DELETED (replaced by tablet e1b42eac370145d4b684456639968485) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:27.942752 15298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "728f8f80b3bb4b10871d8b400f25377b" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
W20260810 08:00:27.943220 14918 leader_election.cc:341] T 114d3eac994f466bb1787e066de05e2c P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
W20260810 08:00:27.944180 15076 leader_election.cc:341] T 728f8f80b3bb4b10871d8b400f25377b P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:27.945317 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a22a48e2ba843b3a32a8900ab937ea0" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224"
I20260810 08:00:27.946020 14987 raft_consensus.cc:3037] T 5a22a48e2ba843b3a32a8900ab937ea0 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:27.934619 15617 leader_election.cc:288] T 5a22a48e2ba843b3a32a8900ab937ea0 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 election: Requested vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:27.950204 15604 leader_election.cc:288] T e46648198f5343f7a65079887e5df159 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:27.953588 14987 raft_consensus.cc:2454] T 5a22a48e2ba843b3a32a8900ab937ea0 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d7219686f304f62a54acc7497cbbd07 in term 1.
I20260810 08:00:27.957222 14988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "728f8f80b3bb4b10871d8b400f25377b" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:27.958592 15437 tablet_bootstrap.cc:492] T 2faab63c5023431c98e7b01ce61e04c2 P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:27.958979 15437 ts_tablet_manager.cc:1397] T 2faab63c5023431c98e7b01ce61e04c2 P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.286s user 0.004s sys 0.007s
I20260810 08:00:27.961066 15437 raft_consensus.cc:348] T 2faab63c5023431c98e7b01ce61e04c2 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:27.963104 15490 tablet_bootstrap.cc:492] T fbf62f4eab8b49f282df3ab4d95d886f P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:27.974905 15437 raft_consensus.cc:374] T 2faab63c5023431c98e7b01ce61e04c2 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:27.975210 15437 raft_consensus.cc:729] T 2faab63c5023431c98e7b01ce61e04c2 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:27.944159 15302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e46648198f5343f7a65079887e5df159" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:27.963568 15299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a22a48e2ba843b3a32a8900ab937ea0" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae"
I20260810 08:00:27.977069 15300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1c4f8f514c94128aedd07bd5efdc3be" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:27.976398 15437 consensus_queue.cc:260] T 2faab63c5023431c98e7b01ce61e04c2 P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:27.999409 15437 ts_tablet_manager.cc:1428] T 2faab63c5023431c98e7b01ce61e04c2 P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.040s user 0.006s sys 0.000s
I20260810 08:00:28.000363 15437 tablet_bootstrap.cc:492] T e46648198f5343f7a65079887e5df159 P a91b27762182454baada71e91ceab224: Bootstrap starting.
W20260810 08:00:28.018297 14692 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.14.57.193:35941 (request call id 31) took 1197 ms (client timeout 1000 ms). Trace:
I20260810 08:00:28.019968 14960 tablet_service.cc:1514] Processing DeleteTablet for tablet 472fbc9204754317aab059a4844d7de1 with delete_type TABLET_DATA_DELETED (replaced by tablet b5a0e0af742340b2945be2102768b307) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:28.024720 14694 master_service.cc:946] ReplaceTablet: received request to replace tablet cb37a3a87f8b48a68c19510f306073c2 from {username='slave'} at 127.0.0.1:42300
I20260810 08:00:28.031677 15482 ts_tablet_manager.cc:1918] T 0c152aab7e244afd857534869b13815c P a91b27762182454baada71e91ceab224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 08:00:28.033686 14691 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_6d487e53be1d46deabb8baa50b2f220e" } new_table_name: "default.Table_e4e38aa2c28146cabb97071119b73368" modify_external_catalogs: true
I20260810 08:00:28.047040 14678 catalog_manager.cc:4841] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): tablet 472fbc9204754317aab059a4844d7de1 (table default.Table_eab0fdadaf49486ca724a0a6e179a889 [id=fc52022e20044cc3bc88ac143e088e69]) successfully deleted
W20260810 08:00:28.061107 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet 2cca8a18778c49d0be01363c31ef180b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2cca8a18778c49d0be01363c31ef180b already in progress: creating tablet
I20260810 08:00:28.065040 14690 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_beb88f8abaa447bfb6962df5718b7313" } modify_external_catalogs: true
I20260810 08:00:28.065795 14690 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_beb88f8abaa447bfb6962df5718b7313" } modify_external_catalogs: true
I20260810 08:00:28.066272 15439 tablet_bootstrap.cc:492] T 35b6da60a5544d7e9b657a5f1e6f90ff P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:28.066695 15439 ts_tablet_manager.cc:1397] T 35b6da60a5544d7e9b657a5f1e6f90ff P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.416s user 0.042s sys 0.063s
W20260810 08:00:28.018751 14692 rpcz_store.cc:269] 0810 08:00:26.820841 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:26.820997 (+ 156us) service_pool.cc:227] Handling call
0810 08:00:26.847852 (+ 26855us) spinlock_profiling.cc:243] Waited 25.3 ms on lock 0x7b5400021554. stack: 00007fe2372c4033 00007fe2372c3f30 00007fe236e45534 00000000004db2b4 00000000004db249 00007fe23e1d0af0 00007fe23e253f2a 00007fe24011fd6c 00007fe240219c3e 00007fe23dd50fb4 00007fe23dd50f42 00007fe23dd50e71 00007fe23dd50dec 00007fe23dd500a2 00007fe23bffa94c 00007fe23bff9d86
0810 08:00:28.018210 (+1170358us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 08:00:27.937517 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 08:00:27.937943 (+ 426us) write_op.cc:432] Acquiring schema lock in shared mode
0810 08:00:27.937978 (+ 35us) write_op.cc:435] Acquired schema lock
0810 08:00:27.937997 (+ 19us) tablet.cc:660] Decoding operations
0810 08:00:27.954726 (+ 16729us) write_op.cc:620] Acquiring the partition lock for write op
0810 08:00:27.954778 (+ 52us) write_op.cc:641] Partition lock acquired for write op
0810 08:00:27.954799 (+ 21us) tablet.cc:683] Acquiring locks for 1 operations
0810 08:00:27.954972 (+ 173us) tablet.cc:699] Row locks acquired
0810 08:00:27.954985 (+ 13us) write_op.cc:260] PREPARE: finished
0810 08:00:27.955104 (+ 119us) write_op.cc:270] Start()
0810 08:00:27.955188 (+ 84us) write_op.cc:276] Timestamp: P: 1786348827955034 usec, L: 0
0810 08:00:27.955205 (+ 17us) op_driver.cc:331] REPLICATION: starting
0810 08:00:27.955707 (+ 502us) log.cc:842] Serialized 532 byte log entry
0810 08:00:27.992133 (+ 36426us) op_driver.cc:444] REPLICATION: finished
0810 08:00:27.993216 (+ 1083us) write_op.cc:301] APPLY: starting
0810 08:00:27.993279 (+ 63us) tablet.cc:1365] starting BulkCheckPresence
0810 08:00:27.993364 (+ 85us) tablet.cc:1368] finished BulkCheckPresence
0810 08:00:27.993522 (+ 158us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 08:00:27.993772 (+ 250us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 08:00:27.993921 (+ 149us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 08:00:27.993962 (+ 41us) write_op.cc:312] APPLY: finished
0810 08:00:27.994173 (+ 211us) log.cc:842] Serialized 22 byte log entry
0810 08:00:27.994308 (+ 135us) write_op.cc:489] Releasing partition, row and schema locks
0810 08:00:27.994363 (+ 55us) write_op.cc:454] Released schema lock
0810 08:00:27.994394 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0810 08:00:27.994588 (+ 194us) write_op.cc:489] Releasing partition, row and schema locks
0810 08:00:27.994612 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":3,"spinlock_wait_cycles":58092416,"child_traces":[["op",{"apply.queue_time_us":907,"apply.run_cpu_time_us":1533,"apply.run_wall_time_us":1575,"num_ops":1,"prepare.queue_time_us":558219,"prepare.run_cpu_time_us":2214,"prepare.run_wall_time_us":24423,"raft.queue_time_us":105,"raft.run_cpu_time_us":870,"raft.run_wall_time_us":869,"replication_time_us":36852,"spinlock_wait_cycles":75008}]]}
I20260810 08:00:28.075992 15439 raft_consensus.cc:348] T 35b6da60a5544d7e9b657a5f1e6f90ff P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.076717 15439 raft_consensus.cc:374] T 35b6da60a5544d7e9b657a5f1e6f90ff P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:28.076988 15439 raft_consensus.cc:729] T 35b6da60a5544d7e9b657a5f1e6f90ff P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:28.074309 14695 master_service.cc:946] ReplaceTablet: received request to replace tablet d22a778bfe784b2f8a7b4d2216909f10 from {username='slave'} at 127.0.0.1:42300
W20260810 08:00:28.079612 14693 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 389b30cba2b149e9ab0de8c191808db2 has no consensus state
W20260810 08:00:28.081516 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet 728f8f80b3bb4b10871d8b400f25377b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 728f8f80b3bb4b10871d8b400f25377b already in progress: creating tablet
W20260810 08:00:28.090137 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet d22a778bfe784b2f8a7b4d2216909f10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d22a778bfe784b2f8a7b4d2216909f10 already in progress: creating tablet
W20260810 08:00:28.091027 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet fbf62f4eab8b49f282df3ab4d95d886f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fbf62f4eab8b49f282df3ab4d95d886f already in progress: creating tablet
I20260810 08:00:28.093838 15482 log.cc:1196] T 0c152aab7e244afd857534869b13815c P a91b27762182454baada71e91ceab224: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal/wals/0c152aab7e244afd857534869b13815c
I20260810 08:00:28.095111 15482 ts_tablet_manager.cc:1939] T 0c152aab7e244afd857534869b13815c P a91b27762182454baada71e91ceab224: Deleting consensus metadata
W20260810 08:00:27.981925 15076 leader_election.cc:341] T 5a22a48e2ba843b3a32a8900ab937ea0 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
W20260810 08:00:28.096941 15076 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.195:37729, user_credentials={real_user=slave}} blocked reactor thread for 115051us
I20260810 08:00:28.097121 15482 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34630 (request call id 52) took 1325 ms. Trace:
W20260810 08:00:28.098129 15076 leader_election.cc:341] T a1c4f8f514c94128aedd07bd5efdc3be P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:28.098829 15437 tablet_bootstrap.cc:654] T e46648198f5343f7a65079887e5df159 P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
W20260810 08:00:28.098738 15076 leader_election.cc:341] T e46648198f5343f7a65079887e5df159 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:28.050475 15479 ts_tablet_manager.cc:1918] T dc3ea02864d040c5bcde38b1a5609676 P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 08:00:28.099326 15479 log.cc:1196] T dc3ea02864d040c5bcde38b1a5609676 P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/dc3ea02864d040c5bcde38b1a5609676
I20260810 08:00:28.100054 15479 ts_tablet_manager.cc:1939] T dc3ea02864d040c5bcde38b1a5609676 P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:28.077608 15439 consensus_queue.cc:260] T 35b6da60a5544d7e9b657a5f1e6f90ff P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.104646 15302 raft_consensus.cc:3037] T 9ddae63b29c34e1397c5765e52e1478f P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:28.105588 15479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43002 (request call id 37) took 2416 ms. Trace:
I20260810 08:00:28.105895 15479 rpcz_store.cc:276] 0810 08:00:25.689287 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:25.689482 (+ 195us) service_pool.cc:227] Handling call
0810 08:00:27.712361 (+2022879us) ts_tablet_manager.cc:1145] Deleting tablet dc3ea02864d040c5bcde38b1a5609676
0810 08:00:27.721889 (+ 9528us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:27.892067 (+170178us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:28.050046 (+157979us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:28.105515 (+ 55469us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":54036,"tablet-delete.queue_time_us":2022022}
I20260810 08:00:28.107126 15479 tablet_replica.cc:330] T 9ddae63b29c34e1397c5765e52e1478f P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:28.107805 15479 raft_consensus.cc:2229] T 9ddae63b29c34e1397c5765e52e1478f P 9d7219686f304f62a54acc7497cbbd07 [term 1 LEADER]: Raft consensus shutting down.
I20260810 08:00:28.108438 15479 pending_rounds.cc:62] T 9ddae63b29c34e1397c5765e52e1478f P 9d7219686f304f62a54acc7497cbbd07: Trying to abort 1 pending ops.
I20260810 08:00:28.108645 15479 pending_rounds.cc:68] T 9ddae63b29c34e1397c5765e52e1478f P 9d7219686f304f62a54acc7497cbbd07: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316884792198668288 op_type: NO_OP noop_request { }
I20260810 08:00:28.109122 15479 raft_consensus.cc:2866] T 9ddae63b29c34e1397c5765e52e1478f P 9d7219686f304f62a54acc7497cbbd07 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260810 08:00:28.109450 15479 raft_consensus.cc:2258] T 9ddae63b29c34e1397c5765e52e1478f P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:28.118729 15479 ts_tablet_manager.cc:1905] T 9ddae63b29c34e1397c5765e52e1478f P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:28.125118 14687 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet d0761b090d8147b8a5537b0ed51108a9
I20260810 08:00:28.126168 15439 ts_tablet_manager.cc:1428] T 35b6da60a5544d7e9b657a5f1e6f90ff P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.059s user 0.007s sys 0.000s
I20260810 08:00:28.127512 15439 tablet_bootstrap.cc:492] T a1c4f8f514c94128aedd07bd5efdc3be P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:28.131681 14960 tablet_service.cc:1514] Processing DeleteTablet for tablet d0761b090d8147b8a5537b0ed51108a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:27 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:28.133873 15280 tablet_service.cc:1514] Processing DeleteTablet for tablet d0761b090d8147b8a5537b0ed51108a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:27 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:28.097419 15482 rpcz_store.cc:276] 0810 08:00:26.772207 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:26.772323 (+ 116us) service_pool.cc:227] Handling call
0810 08:00:27.724303 (+951980us) ts_tablet_manager.cc:1145] Deleting tablet 0c152aab7e244afd857534869b13815c
0810 08:00:27.724337 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:27.953190 (+228853us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:28.031444 (+ 78254us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:28.097057 (+ 65613us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":911046}
W20260810 08:00:28.135888 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet d0761b090d8147b8a5537b0ed51108a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0761b090d8147b8a5537b0ed51108a9 already in progress: creating tablet
I20260810 08:00:28.136777 14681 catalog_manager.cc:4841] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): tablet 0c152aab7e244afd857534869b13815c (table default.Table_8ac532f4969e45a197db311de2a7b795 [id=7893824f2c374508b7b3c4635ebea5b6]) successfully deleted
W20260810 08:00:28.146104 14681 catalog_manager.cc:4821] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet 2cca8a18778c49d0be01363c31ef180b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2cca8a18778c49d0be01363c31ef180b already in progress: creating tablet
I20260810 08:00:28.152194 15437 tablet_bootstrap.cc:492] T e46648198f5343f7a65079887e5df159 P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:28.152709 15437 ts_tablet_manager.cc:1397] T e46648198f5343f7a65079887e5df159 P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.153s user 0.009s sys 0.001s
W20260810 08:00:28.153962 14681 catalog_manager.cc:4821] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet 728f8f80b3bb4b10871d8b400f25377b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 728f8f80b3bb4b10871d8b400f25377b already in progress: creating tablet
W20260810 08:00:28.155485 14681 catalog_manager.cc:4821] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet d22a778bfe784b2f8a7b4d2216909f10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d22a778bfe784b2f8a7b4d2216909f10 already in progress: creating tablet
W20260810 08:00:28.159659 14681 catalog_manager.cc:4821] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet fbf62f4eab8b49f282df3ab4d95d886f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fbf62f4eab8b49f282df3ab4d95d886f already in progress: creating tablet
W20260810 08:00:28.160341 14681 catalog_manager.cc:4808] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet 472fbc9204754317aab059a4844d7de1 because the tablet was not found. No further retry: Not found: Tablet not found: 472fbc9204754317aab059a4844d7de1
W20260810 08:00:28.161006 14681 catalog_manager.cc:4821] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet d0761b090d8147b8a5537b0ed51108a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0761b090d8147b8a5537b0ed51108a9 already in progress: creating tablet
W20260810 08:00:28.171494 14691 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3ac554bd37b045a2a8cf2743f4645f28 has no consensus state
W20260810 08:00:28.170631 15079 consensus_peers.cc:487] T 0c152aab7e244afd857534869b13815c P 9d7219686f304f62a54acc7497cbbd07 -> Peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Couldn't send request to peer a91b27762182454baada71e91ceab224. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260810 08:00:28.173380 15079 leader_election.cc:302] T a1c4f8f514c94128aedd07bd5efdc3be P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224; no voters: 3fc5e568f3724f158b616c303efb9eae
I20260810 08:00:28.174413 15623 raft_consensus.cc:2781] T a1c4f8f514c94128aedd07bd5efdc3be P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:28.174223 15437 raft_consensus.cc:348] T e46648198f5343f7a65079887e5df159 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.174742 15623 raft_consensus.cc:482] T a1c4f8f514c94128aedd07bd5efdc3be P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260810 08:00:28.174795 15079 leader_election.cc:341] T e46648198f5343f7a65079887e5df159 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:28.175113 15437 raft_consensus.cc:374] T e46648198f5343f7a65079887e5df159 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:28.175139 15623 raft_consensus.cc:3037] T a1c4f8f514c94128aedd07bd5efdc3be P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:28.175474 15437 raft_consensus.cc:729] T e46648198f5343f7a65079887e5df159 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:28.176282 15437 consensus_queue.cc:260] T e46648198f5343f7a65079887e5df159 P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.175248 15079 leader_election.cc:302] T e46648198f5343f7a65079887e5df159 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:28.178483 15604 raft_consensus.cc:2726] T e46648198f5343f7a65079887e5df159 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:28.183162 14694 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet cb37a3a87f8b48a68c19510f306073c2
I20260810 08:00:28.184446 14694 catalog_manager.cc:6565] ReplaceTablet: tablet cb37a3a87f8b48a68c19510f306073c2 deleted and replaced by tablet b18980bffe6045b5b69bcecb47b4c711
I20260810 08:00:28.179571 15079 leader_election.cc:302] T 5a22a48e2ba843b3a32a8900ab937ea0 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224; no voters: 3fc5e568f3724f158b616c303efb9eae
I20260810 08:00:28.185564 15604 raft_consensus.cc:2781] T 5a22a48e2ba843b3a32a8900ab937ea0 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 08:00:28.186038 15604 raft_consensus.cc:686] T 5a22a48e2ba843b3a32a8900ab937ea0 P 9d7219686f304f62a54acc7497cbbd07 [term 1 LEADER]: Becoming Leader. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Running, Role: LEADER
I20260810 08:00:28.186625 15490 tablet_bootstrap.cc:654] T fbf62f4eab8b49f282df3ab4d95d886f P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:28.186861 15604 consensus_queue.cc:237] T 5a22a48e2ba843b3a32a8900ab937ea0 P 9d7219686f304f62a54acc7497cbbd07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.191313 14960 tablet_service.cc:1514] Processing DeleteTablet for tablet cb37a3a87f8b48a68c19510f306073c2 with delete_type TABLET_DATA_DELETED (replaced by tablet b18980bffe6045b5b69bcecb47b4c711) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:28.191799 15439 tablet_bootstrap.cc:654] T a1c4f8f514c94128aedd07bd5efdc3be P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:28.191531 15280 tablet_service.cc:1514] Processing DeleteTablet for tablet cb37a3a87f8b48a68c19510f306073c2 with delete_type TABLET_DATA_DELETED (replaced by tablet b18980bffe6045b5b69bcecb47b4c711) from {username='slave'} at 127.0.0.1:40692
W20260810 08:00:28.193300 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet cb37a3a87f8b48a68c19510f306073c2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb37a3a87f8b48a68c19510f306073c2 already in progress: creating tablet
I20260810 08:00:28.194926 15623 raft_consensus.cc:504] T a1c4f8f514c94128aedd07bd5efdc3be P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.205276 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1c4f8f514c94128aedd07bd5efdc3be" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae"
W20260810 08:00:28.201561 15079 leader_election.cc:341] T 728f8f80b3bb4b10871d8b400f25377b P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
W20260810 08:00:28.206899 15076 leader_election.cc:341] T a1c4f8f514c94128aedd07bd5efdc3be P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:28.207039 15079 leader_election.cc:302] T 728f8f80b3bb4b10871d8b400f25377b P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:28.209955 15617 raft_consensus.cc:2726] T 728f8f80b3bb4b10871d8b400f25377b P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 08:00:28.211467 14681 catalog_manager.cc:4821] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet cb37a3a87f8b48a68c19510f306073c2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb37a3a87f8b48a68c19510f306073c2 already in progress: creating tablet
I20260810 08:00:28.212123 15437 ts_tablet_manager.cc:1428] T e46648198f5343f7a65079887e5df159 P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.059s user 0.005s sys 0.000s
I20260810 08:00:28.213426 15437 tablet_bootstrap.cc:492] T 0fae4809ef5e4a9fb16976674e287f03 P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:28.217048 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1c4f8f514c94128aedd07bd5efdc3be" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224"
I20260810 08:00:28.217697 14987 raft_consensus.cc:3037] T a1c4f8f514c94128aedd07bd5efdc3be P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
W20260810 08:00:28.218223 14922 leader_election.cc:341] T 114d3eac994f466bb1787e066de05e2c P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Not found: Tablet not found: 114d3eac994f466bb1787e066de05e2c
I20260810 08:00:28.218699 15623 leader_election.cc:288] T a1c4f8f514c94128aedd07bd5efdc3be P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 election: Requested vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:28.218648 14922 leader_election.cc:302] T 114d3eac994f466bb1787e066de05e2c P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a91b27762182454baada71e91ceab224; no voters: 3fc5e568f3724f158b616c303efb9eae, 9d7219686f304f62a54acc7497cbbd07
I20260810 08:00:28.220089 15443 raft_consensus.cc:2726] T 114d3eac994f466bb1787e066de05e2c P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:28.227535 14691 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_fea0f565d8a0450a91e8729141f89d99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:28.230343 14987 raft_consensus.cc:2454] T a1c4f8f514c94128aedd07bd5efdc3be P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d7219686f304f62a54acc7497cbbd07 in term 1.
I20260810 08:00:28.230854 15439 tablet_bootstrap.cc:492] T a1c4f8f514c94128aedd07bd5efdc3be P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:28.231277 15439 ts_tablet_manager.cc:1397] T a1c4f8f514c94128aedd07bd5efdc3be P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.104s user 0.008s sys 0.000s
I20260810 08:00:28.232854 15437 tablet_bootstrap.cc:654] T 0fae4809ef5e4a9fb16976674e287f03 P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:28.233466 15439 raft_consensus.cc:348] T a1c4f8f514c94128aedd07bd5efdc3be P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.234170 15439 raft_consensus.cc:374] T a1c4f8f514c94128aedd07bd5efdc3be P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:28.234436 15439 raft_consensus.cc:729] T a1c4f8f514c94128aedd07bd5efdc3be P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
W20260810 08:00:28.236590 14691 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fea0f565d8a0450a91e8729141f89d99 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:28.237920 15439 consensus_queue.cc:260] T a1c4f8f514c94128aedd07bd5efdc3be P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.246721 15117 tablet_service.cc:1514] Processing DeleteTablet for tablet d0761b090d8147b8a5537b0ed51108a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:27 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:28.249013 15117 tablet_service.cc:1514] Processing DeleteTablet for tablet cb37a3a87f8b48a68c19510f306073c2 with delete_type TABLET_DATA_DELETED (replaced by tablet b18980bffe6045b5b69bcecb47b4c711) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:28.250051 15079 leader_election.cc:302] T a1c4f8f514c94128aedd07bd5efdc3be P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224; no voters: 3fc5e568f3724f158b616c303efb9eae
I20260810 08:00:28.250811 15439 ts_tablet_manager.cc:1428] T a1c4f8f514c94128aedd07bd5efdc3be P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.019s user 0.006s sys 0.001s
I20260810 08:00:28.251451 15604 raft_consensus.cc:2781] T a1c4f8f514c94128aedd07bd5efdc3be P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 08:00:28.251955 15604 raft_consensus.cc:686] T a1c4f8f514c94128aedd07bd5efdc3be P 9d7219686f304f62a54acc7497cbbd07 [term 1 LEADER]: Becoming Leader. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Running, Role: LEADER
I20260810 08:00:28.252841 15490 tablet_bootstrap.cc:492] T fbf62f4eab8b49f282df3ab4d95d886f P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:28.253249 15490 ts_tablet_manager.cc:1397] T fbf62f4eab8b49f282df3ab4d95d886f P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.290s user 0.008s sys 0.002s
I20260810 08:00:28.252760 15604 consensus_queue.cc:237] T a1c4f8f514c94128aedd07bd5efdc3be P 9d7219686f304f62a54acc7497cbbd07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.256407 15490 raft_consensus.cc:348] T fbf62f4eab8b49f282df3ab4d95d886f P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:28.257205 15490 raft_consensus.cc:374] T fbf62f4eab8b49f282df3ab4d95d886f P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:28.257519 15490 raft_consensus.cc:729] T fbf62f4eab8b49f282df3ab4d95d886f P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:28.258225 15490 consensus_queue.cc:260] T fbf62f4eab8b49f282df3ab4d95d886f P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:28.264017 15439 tablet_bootstrap.cc:492] T 114d3eac994f466bb1787e066de05e2c P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:28.269073 14691 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_ebe6adac96b74785b939b2a0d8e7a192"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:28.272843 14922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.57.194:36265, user_credentials={real_user=slave}} blocked reactor thread for 52846.1us
I20260810 08:00:28.277541 14686 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_4f0bc756581c45958dc46729a9430f79"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:28.279460 14686 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f0bc756581c45958dc46729a9430f79 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 08:00:28.274751 14691 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ebe6adac96b74785b939b2a0d8e7a192 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:28.285002 15439 tablet_bootstrap.cc:654] T 114d3eac994f466bb1787e066de05e2c P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:28.291602 15479 ts_tablet_manager.cc:1918] T 9ddae63b29c34e1397c5765e52e1478f P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 08:00:28.292145 15479 log.cc:1196] T 9ddae63b29c34e1397c5765e52e1478f P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/9ddae63b29c34e1397c5765e52e1478f
I20260810 08:00:28.293694 15479 ts_tablet_manager.cc:1939] T 9ddae63b29c34e1397c5765e52e1478f P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:28.296030 15479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43002 (request call id 40) took 2029 ms. Trace:
I20260810 08:00:28.297083 15479 rpcz_store.cc:276] 0810 08:00:26.269762 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:26.271773 (+ 2011us) service_pool.cc:227] Handling call
0810 08:00:28.106963 (+1835190us) ts_tablet_manager.cc:1145] Deleting tablet 9ddae63b29c34e1397c5765e52e1478f
0810 08:00:28.107022 (+ 59us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:28.218368 (+111346us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:28.291347 (+ 72979us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:28.295974 (+ 4627us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1834580}
I20260810 08:00:28.312206 15490 ts_tablet_manager.cc:1428] T fbf62f4eab8b49f282df3ab4d95d886f P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.059s user 0.003s sys 0.005s
I20260810 08:00:28.313715 15479 tablet_replica.cc:330] T 472fbc9204754317aab059a4844d7de1 P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:28.313926 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet 9ddae63b29c34e1397c5765e52e1478f (table default.Table_0e30c621046242609c9f6fc73b16d0e7 [id=e9608d66a9964bc6aafdf84f574fba02]) successfully deleted
I20260810 08:00:28.313970 14693 master_service.cc:946] ReplaceTablet: received request to replace tablet 37d8a37a16f34a0285c43c0d71b5f072 from {username='slave'} at 127.0.0.1:42300
I20260810 08:00:28.314760 15490 tablet_bootstrap.cc:492] T 49c35b7f4a72437fa0675abe8a8f3a71 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:28.315555 15479 raft_consensus.cc:2229] T 472fbc9204754317aab059a4844d7de1 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:28.324843 15479 raft_consensus.cc:2258] T 472fbc9204754317aab059a4844d7de1 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:28.327052 15479 ts_tablet_manager.cc:1905] T 472fbc9204754317aab059a4844d7de1 P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:28.330070 15439 tablet_bootstrap.cc:492] T 114d3eac994f466bb1787e066de05e2c P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:28.338860 14695 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_44d94cfdac894ff68cfbb8984912231c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:28.340426 14695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44d94cfdac894ff68cfbb8984912231c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:28.341828 15439 ts_tablet_manager.cc:1397] T 114d3eac994f466bb1787e066de05e2c P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.078s user 0.009s sys 0.002s
I20260810 08:00:28.345557 15439 raft_consensus.cc:348] T 114d3eac994f466bb1787e066de05e2c P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.346403 15439 raft_consensus.cc:374] T 114d3eac994f466bb1787e066de05e2c P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:28.346652 15439 raft_consensus.cc:729] T 114d3eac994f466bb1787e066de05e2c P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:28.333207 15490 tablet_bootstrap.cc:654] T 49c35b7f4a72437fa0675abe8a8f3a71 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:28.346882 14686 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_d7227a2f586b4fd29babdc0ae25a0328" } modify_external_catalogs: true
I20260810 08:00:28.347435 14686 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_d7227a2f586b4fd29babdc0ae25a0328" } modify_external_catalogs: true
I20260810 08:00:28.351903 14692 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_dbbc4dceb17d4aecb41b3ec5fc8c21ed" } new_table_name: "default.Table_d81fe7cc77ad45d39c8b44e1299dbefc" modify_external_catalogs: true
I20260810 08:00:28.356678 15439 consensus_queue.cc:260] T 114d3eac994f466bb1787e066de05e2c P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.372337 15439 ts_tablet_manager.cc:1428] T 114d3eac994f466bb1787e066de05e2c P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.030s user 0.006s sys 0.000s
I20260810 08:00:28.377141 14691 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_5e1d024b54504287840e8c45c056fa56" } new_table_name: "default.Table_4f3d374da9fd4fcaaff35b3e4c3bdbf7" modify_external_catalogs: true
I20260810 08:00:28.377679 15439 tablet_bootstrap.cc:492] T 5a22a48e2ba843b3a32a8900ab937ea0 P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
W20260810 08:00:28.384979 14692 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e46648198f5343f7a65079887e5df159 consensus state has no leader
I20260810 08:00:28.403959 15490 tablet_bootstrap.cc:492] T 49c35b7f4a72437fa0675abe8a8f3a71 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:28.404489 15490 ts_tablet_manager.cc:1397] T 49c35b7f4a72437fa0675abe8a8f3a71 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.090s user 0.009s sys 0.001s
I20260810 08:00:28.406688 15490 raft_consensus.cc:348] T 49c35b7f4a72437fa0675abe8a8f3a71 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.407378 15490 raft_consensus.cc:374] T 49c35b7f4a72437fa0675abe8a8f3a71 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:28.407598 15490 raft_consensus.cc:729] T 49c35b7f4a72437fa0675abe8a8f3a71 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
W20260810 08:00:28.413512 14691 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4381ccb8711744ff8f9766ef49c53e28 consensus state has no leader
I20260810 08:00:28.415508 15439 tablet_bootstrap.cc:654] T 5a22a48e2ba843b3a32a8900ab937ea0 P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:28.418715 15604 raft_consensus.cc:482] T 0fae4809ef5e4a9fb16976674e287f03 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:28.424348 15604 raft_consensus.cc:504] T 0fae4809ef5e4a9fb16976674e287f03 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.425585 14692 master_service.cc:946] ReplaceTablet: received request to replace tablet a1c4f8f514c94128aedd07bd5efdc3be from {username='slave'} at 127.0.0.1:42300
I20260810 08:00:28.426602 15604 leader_election.cc:288] T 0fae4809ef5e4a9fb16976674e287f03 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:28.427518 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fae4809ef5e4a9fb16976674e287f03" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:28.427750 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fae4809ef5e4a9fb16976674e287f03" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
W20260810 08:00:28.428951 15079 leader_election.cc:341] T 0fae4809ef5e4a9fb16976674e287f03 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:28.414016 15490 consensus_queue.cc:260] T 49c35b7f4a72437fa0675abe8a8f3a71 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
W20260810 08:00:28.429562 15076 leader_election.cc:341] T 0fae4809ef5e4a9fb16976674e287f03 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:28.429986 15076 leader_election.cc:302] T 0fae4809ef5e4a9fb16976674e287f03 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:28.430962 15490 ts_tablet_manager.cc:1428] T 49c35b7f4a72437fa0675abe8a8f3a71 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.026s user 0.004s sys 0.003s
I20260810 08:00:28.431475 15623 raft_consensus.cc:2726] T 0fae4809ef5e4a9fb16976674e287f03 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:28.432188 15490 tablet_bootstrap.cc:492] T 2cca8a18778c49d0be01363c31ef180b P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:28.437247 15490 tablet_bootstrap.cc:654] T 2cca8a18778c49d0be01363c31ef180b P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:28.413604 15479 ts_tablet_manager.cc:1918] T 472fbc9204754317aab059a4844d7de1 P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:28.442409 15479 log.cc:1196] T 472fbc9204754317aab059a4844d7de1 P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/472fbc9204754317aab059a4844d7de1
I20260810 08:00:28.442655 15437 tablet_bootstrap.cc:492] T 0fae4809ef5e4a9fb16976674e287f03 P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:28.443066 15437 ts_tablet_manager.cc:1397] T 0fae4809ef5e4a9fb16976674e287f03 P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.230s user 0.009s sys 0.000s
I20260810 08:00:28.443750 15479 ts_tablet_manager.cc:1939] T 472fbc9204754317aab059a4844d7de1 P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:28.446359 15437 raft_consensus.cc:348] T 0fae4809ef5e4a9fb16976674e287f03 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.455845 15439 tablet_bootstrap.cc:492] T 5a22a48e2ba843b3a32a8900ab937ea0 P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:28.456419 15439 ts_tablet_manager.cc:1397] T 5a22a48e2ba843b3a32a8900ab937ea0 P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.079s user 0.011s sys 0.000s
I20260810 08:00:28.458968 14695 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_e6beb01e821446feaced3468aaa20563"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:28.458968 15439 raft_consensus.cc:348] T 5a22a48e2ba843b3a32a8900ab937ea0 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.459653 15439 raft_consensus.cc:374] T 5a22a48e2ba843b3a32a8900ab937ea0 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:28.459915 15439 raft_consensus.cc:729] T 5a22a48e2ba843b3a32a8900ab937ea0 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
W20260810 08:00:28.460762 14695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6beb01e821446feaced3468aaa20563 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:28.460549 15439 consensus_queue.cc:260] T 5a22a48e2ba843b3a32a8900ab937ea0 P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.461498 15479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43002 (request call id 41) took 2187 ms. Trace:
I20260810 08:00:28.467960 15479 rpcz_store.cc:276] 0810 08:00:26.276604 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:26.281834 (+ 5230us) service_pool.cc:227] Handling call
0810 08:00:28.298087 (+2016253us) ts_tablet_manager.cc:1145] Deleting tablet 472fbc9204754317aab059a4844d7de1
0810 08:00:28.313552 (+ 15465us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:28.382139 (+ 68587us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:28.413297 (+ 31158us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:28.449592 (+ 36295us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2015665}
I20260810 08:00:28.470353 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet 472fbc9204754317aab059a4844d7de1 (table default.Table_eab0fdadaf49486ca724a0a6e179a889 [id=fc52022e20044cc3bc88ac143e088e69]) successfully deleted
I20260810 08:00:28.470885 15479 tablet_replica.cc:330] T 37d8a37a16f34a0285c43c0d71b5f072 P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:28.478243 15479 raft_consensus.cc:2229] T 37d8a37a16f34a0285c43c0d71b5f072 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:28.484788 15479 raft_consensus.cc:2258] T 37d8a37a16f34a0285c43c0d71b5f072 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:28.486474 15439 ts_tablet_manager.cc:1428] T 5a22a48e2ba843b3a32a8900ab937ea0 P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.030s user 0.006s sys 0.000s
I20260810 08:00:28.457103 15437 raft_consensus.cc:374] T 0fae4809ef5e4a9fb16976674e287f03 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:28.488811 15437 raft_consensus.cc:729] T 0fae4809ef5e4a9fb16976674e287f03 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:28.491133 15479 ts_tablet_manager.cc:1905] T 37d8a37a16f34a0285c43c0d71b5f072 P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:28.498322 15439 tablet_bootstrap.cc:492] T e46648198f5343f7a65079887e5df159 P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:28.489513 15437 consensus_queue.cc:260] T 0fae4809ef5e4a9fb16976674e287f03 P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.500725 15437 ts_tablet_manager.cc:1428] T 0fae4809ef5e4a9fb16976674e287f03 P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.057s user 0.008s sys 0.000s
I20260810 08:00:28.510084 15631 raft_consensus.cc:482] T 2faab63c5023431c98e7b01ce61e04c2 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:28.510576 15631 raft_consensus.cc:504] T 2faab63c5023431c98e7b01ce61e04c2 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.512475 15631 leader_election.cc:288] T 2faab63c5023431c98e7b01ce61e04c2 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:28.516094 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2faab63c5023431c98e7b01ce61e04c2" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:28.516785 15301 raft_consensus.cc:2454] T 2faab63c5023431c98e7b01ce61e04c2 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 0.
W20260810 08:00:28.518395 14922 leader_election.cc:341] T 2faab63c5023431c98e7b01ce61e04c2 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Not found: Tablet not found: 2faab63c5023431c98e7b01ce61e04c2
I20260810 08:00:28.520699 14918 leader_election.cc:302] T 2faab63c5023431c98e7b01ce61e04c2 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224; no voters: 9d7219686f304f62a54acc7497cbbd07
I20260810 08:00:28.522039 15631 raft_consensus.cc:2781] T 2faab63c5023431c98e7b01ce61e04c2 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:28.522449 15631 raft_consensus.cc:482] T 2faab63c5023431c98e7b01ce61e04c2 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:28.522826 15631 raft_consensus.cc:3037] T 2faab63c5023431c98e7b01ce61e04c2 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:28.517539 15439 tablet_bootstrap.cc:654] T e46648198f5343f7a65079887e5df159 P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:28.534952 15631 raft_consensus.cc:504] T 2faab63c5023431c98e7b01ce61e04c2 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.542611 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2faab63c5023431c98e7b01ce61e04c2" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae"
I20260810 08:00:28.543272 15301 raft_consensus.cc:3037] T 2faab63c5023431c98e7b01ce61e04c2 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:28.547991 15437 tablet_bootstrap.cc:492] T 728f8f80b3bb4b10871d8b400f25377b P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:28.560117 15276 tablet_service.cc:1467] Processing CreateTablet for tablet a244ad3bda5d45f9b89f68b734338d21 (DEFAULT_TABLE table=default.Table_1782c242ad164aa69cb679bbb072934e [id=5efc99cbf96549d681a989f1886b65f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:28.561232 15276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a244ad3bda5d45f9b89f68b734338d21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:28.567370 15490 tablet_bootstrap.cc:492] T 2cca8a18778c49d0be01363c31ef180b P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:28.567934 15490 ts_tablet_manager.cc:1397] T 2cca8a18778c49d0be01363c31ef180b P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.136s user 0.011s sys 0.006s
I20260810 08:00:28.570423 14691 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_357e1765e09e4c4ab104ee0436bc1091"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:28.572197 14691 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_357e1765e09e4c4ab104ee0436bc1091 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:28.572266 15490 raft_consensus.cc:348] T 2cca8a18778c49d0be01363c31ef180b P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.573156 15490 raft_consensus.cc:374] T 2cca8a18778c49d0be01363c31ef180b P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:28.573447 15490 raft_consensus.cc:729] T 2cca8a18778c49d0be01363c31ef180b P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:28.577811 15439 tablet_bootstrap.cc:492] T e46648198f5343f7a65079887e5df159 P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:28.578267 15439 ts_tablet_manager.cc:1397] T e46648198f5343f7a65079887e5df159 P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.085s user 0.008s sys 0.003s
I20260810 08:00:28.580605 15301 raft_consensus.cc:2454] T 2faab63c5023431c98e7b01ce61e04c2 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 1.
I20260810 08:00:28.581243 15490 consensus_queue.cc:260] T 2cca8a18778c49d0be01363c31ef180b P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.581990 14918 leader_election.cc:302] T 2faab63c5023431c98e7b01ce61e04c2 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224; no voters:
I20260810 08:00:28.582929 15640 raft_consensus.cc:2781] T 2faab63c5023431c98e7b01ce61e04c2 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 08:00:28.583398 15640 raft_consensus.cc:686] T 2faab63c5023431c98e7b01ce61e04c2 P a91b27762182454baada71e91ceab224 [term 1 LEADER]: Becoming Leader. State: Replica: a91b27762182454baada71e91ceab224, State: Running, Role: LEADER
I20260810 08:00:28.584188 15640 consensus_queue.cc:237] T 2faab63c5023431c98e7b01ce61e04c2 P a91b27762182454baada71e91ceab224 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.585788 14693 master_service.cc:946] ReplaceTablet: received request to replace tablet 924c0f73638f48a88c08d45d459132e1 from {username='slave'} at 127.0.0.1:42300
I20260810 08:00:28.594831 14695 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_ae0fee60b7704372bf9ac04a22df6876"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:28.598034 14695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae0fee60b7704372bf9ac04a22df6876 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:28.601748 15490 ts_tablet_manager.cc:1428] T 2cca8a18778c49d0be01363c31ef180b P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.033s user 0.008s sys 0.000s
I20260810 08:00:28.580368 15439 raft_consensus.cc:348] T e46648198f5343f7a65079887e5df159 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.612164 15439 raft_consensus.cc:374] T e46648198f5343f7a65079887e5df159 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:28.612476 15439 raft_consensus.cc:729] T e46648198f5343f7a65079887e5df159 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:28.612347 14960 tablet_service.cc:1467] Processing CreateTablet for tablet a244ad3bda5d45f9b89f68b734338d21 (DEFAULT_TABLE table=default.Table_1782c242ad164aa69cb679bbb072934e [id=5efc99cbf96549d681a989f1886b65f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:28.610726 15490 tablet_bootstrap.cc:492] T 25de7bf64bf6450ba17b7748568bc0ed P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:28.613533 14960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a244ad3bda5d45f9b89f68b734338d21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:28.615296 15439 consensus_queue.cc:260] T e46648198f5343f7a65079887e5df159 P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.623386 15117 tablet_service.cc:1467] Processing CreateTablet for tablet a244ad3bda5d45f9b89f68b734338d21 (DEFAULT_TABLE table=default.Table_1782c242ad164aa69cb679bbb072934e [id=5efc99cbf96549d681a989f1886b65f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:28.625798 15447 raft_consensus.cc:482] T 35b6da60a5544d7e9b657a5f1e6f90ff P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:28.626235 15447 raft_consensus.cc:504] T 35b6da60a5544d7e9b657a5f1e6f90ff P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.628523 15447 leader_election.cc:288] T 35b6da60a5544d7e9b657a5f1e6f90ff P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:28.630748 14690 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet 0fae4809ef5e4a9fb16976674e287f03
I20260810 08:00:28.631349 15439 ts_tablet_manager.cc:1428] T e46648198f5343f7a65079887e5df159 P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.053s user 0.004s sys 0.002s
I20260810 08:00:28.631304 15147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b6da60a5544d7e9b657a5f1e6f90ff" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:28.631940 15147 raft_consensus.cc:2454] T 35b6da60a5544d7e9b657a5f1e6f90ff P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 0.
I20260810 08:00:28.632893 15640 raft_consensus.cc:482] T e46648198f5343f7a65079887e5df159 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:28.632848 15234 leader_election.cc:302] T 35b6da60a5544d7e9b657a5f1e6f90ff P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, 9d7219686f304f62a54acc7497cbbd07; no voters:
I20260810 08:00:28.633366 15640 raft_consensus.cc:504] T e46648198f5343f7a65079887e5df159 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.634217 15280 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fae4809ef5e4a9fb16976674e287f03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:28 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:28.635366 15640 leader_election.cc:288] T e46648198f5343f7a65079887e5df159 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
W20260810 08:00:28.635469 15234 leader_election.cc:341] T 35b6da60a5544d7e9b657a5f1e6f90ff P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Not found: Tablet not found: 35b6da60a5544d7e9b657a5f1e6f90ff
I20260810 08:00:28.636082 15300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e46648198f5343f7a65079887e5df159" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
W20260810 08:00:28.636439 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet 0fae4809ef5e4a9fb16976674e287f03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0fae4809ef5e4a9fb16976674e287f03 already in progress: creating tablet
I20260810 08:00:28.636569 15447 raft_consensus.cc:2781] T 35b6da60a5544d7e9b657a5f1e6f90ff P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:28.636674 15300 raft_consensus.cc:2454] T e46648198f5343f7a65079887e5df159 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 0.
I20260810 08:00:28.637058 15447 raft_consensus.cc:482] T 35b6da60a5544d7e9b657a5f1e6f90ff P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:28.637465 15447 raft_consensus.cc:3037] T 35b6da60a5544d7e9b657a5f1e6f90ff P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:28.637687 15125 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fae4809ef5e4a9fb16976674e287f03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:28 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:28.638055 14918 leader_election.cc:302] T e46648198f5343f7a65079887e5df159 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224; no voters:
I20260810 08:00:28.638418 14962 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fae4809ef5e4a9fb16976674e287f03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:28 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:28.638813 15640 raft_consensus.cc:2781] T e46648198f5343f7a65079887e5df159 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:28.639104 15482 tablet_replica.cc:330] T 0fae4809ef5e4a9fb16976674e287f03 P a91b27762182454baada71e91ceab224: stopping tablet replica
I20260810 08:00:28.639140 15640 raft_consensus.cc:482] T e46648198f5343f7a65079887e5df159 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:28.639530 15640 raft_consensus.cc:3037] T e46648198f5343f7a65079887e5df159 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:28.639865 15482 raft_consensus.cc:2229] T 0fae4809ef5e4a9fb16976674e287f03 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:28.640319 15482 raft_consensus.cc:2258] T 0fae4809ef5e4a9fb16976674e287f03 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:28.643498 15482 ts_tablet_manager.cc:1905] T 0fae4809ef5e4a9fb16976674e287f03 P a91b27762182454baada71e91ceab224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:28.646015 15117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a244ad3bda5d45f9b89f68b734338d21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:28.646083 15479 ts_tablet_manager.cc:1918] T 37d8a37a16f34a0285c43c0d71b5f072 P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:28.646430 15479 log.cc:1196] T 37d8a37a16f34a0285c43c0d71b5f072 P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/37d8a37a16f34a0285c43c0d71b5f072
I20260810 08:00:28.646569 15147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e46648198f5343f7a65079887e5df159" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:28.647239 15147 raft_consensus.cc:2454] T e46648198f5343f7a65079887e5df159 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 0.
I20260810 08:00:28.651790 15617 raft_consensus.cc:482] T 2cca8a18778c49d0be01363c31ef180b P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:28.652235 15617 raft_consensus.cc:504] T 2cca8a18778c49d0be01363c31ef180b P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.654166 15617 leader_election.cc:288] T 2cca8a18778c49d0be01363c31ef180b P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), a91b27762182454baada71e91ceab224 (127.14.57.193:43601)
I20260810 08:00:28.654770 14690 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_f3dd0fbbcd664605b91fb032e4877812" } new_table_name: "default.Table_9d3325c7496340d2bf7e36c9618fd581" modify_external_catalogs: true
I20260810 08:00:28.655784 15300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cca8a18778c49d0be01363c31ef180b" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
W20260810 08:00:28.657114 15076 leader_election.cc:341] T 2cca8a18778c49d0be01363c31ef180b P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:28.658829 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cca8a18778c49d0be01363c31ef180b" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
W20260810 08:00:28.660121 15079 leader_election.cc:341] T 2cca8a18778c49d0be01363c31ef180b P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:28.660497 15079 leader_election.cc:302] T 2cca8a18778c49d0be01363c31ef180b P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:28.661306 15617 raft_consensus.cc:2726] T 2cca8a18778c49d0be01363c31ef180b P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:28.667950 15439 tablet_bootstrap.cc:492] T 0fae4809ef5e4a9fb16976674e287f03 P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:28.674988 15640 raft_consensus.cc:504] T e46648198f5343f7a65079887e5df159 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.676646 15631 leader_election.cc:288] T 2faab63c5023431c98e7b01ce61e04c2 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Requested vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:28.679802 15300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e46648198f5343f7a65079887e5df159" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae"
I20260810 08:00:28.680440 15300 raft_consensus.cc:3037] T e46648198f5343f7a65079887e5df159 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Advancing to term 1
W20260810 08:00:28.680545 14922 leader_election.cc:341] T 2faab63c5023431c98e7b01ce61e04c2 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Not found: Tablet not found: 2faab63c5023431c98e7b01ce61e04c2
I20260810 08:00:28.682252 15479 ts_tablet_manager.cc:1939] T 37d8a37a16f34a0285c43c0d71b5f072 P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:28.685061 15640 leader_election.cc:288] T e46648198f5343f7a65079887e5df159 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Requested vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:28.685423 15147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e46648198f5343f7a65079887e5df159" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07"
I20260810 08:00:28.686120 15147 raft_consensus.cc:3037] T e46648198f5343f7a65079887e5df159 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:28.689109 14962 tablet_service.cc:1467] Processing CreateTablet for tablet 8364045e3fa2441e8dc1a1d87f73361d (DEFAULT_TABLE table=default.Table_f3dd0fbbcd664605b91fb032e4877812 [id=97abcf4bf320496ebbec1fd2ed20765b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:28.690249 14962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8364045e3fa2441e8dc1a1d87f73361d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:28.702193 15280 tablet_service.cc:1467] Processing CreateTablet for tablet 8364045e3fa2441e8dc1a1d87f73361d (DEFAULT_TABLE table=default.Table_f3dd0fbbcd664605b91fb032e4877812 [id=97abcf4bf320496ebbec1fd2ed20765b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:28.703860 15280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8364045e3fa2441e8dc1a1d87f73361d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:28.707566 15437 tablet_bootstrap.cc:654] T 728f8f80b3bb4b10871d8b400f25377b P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:28.708624 15617 raft_consensus.cc:482] T 49c35b7f4a72437fa0675abe8a8f3a71 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:28.709162 15617 raft_consensus.cc:504] T 49c35b7f4a72437fa0675abe8a8f3a71 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.712607 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49c35b7f4a72437fa0675abe8a8f3a71" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
W20260810 08:00:28.713963 15076 leader_election.cc:341] T 49c35b7f4a72437fa0675abe8a8f3a71 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:28.714787 15617 leader_election.cc:288] T 49c35b7f4a72437fa0675abe8a8f3a71 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:28.716666 14692 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet a1c4f8f514c94128aedd07bd5efdc3be
I20260810 08:00:28.718222 14692 catalog_manager.cc:6565] ReplaceTablet: tablet a1c4f8f514c94128aedd07bd5efdc3be deleted and replaced by tablet a4a78b2a960d4d78aa1f48a9c9edf3be
I20260810 08:00:28.718716 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49c35b7f4a72437fa0675abe8a8f3a71" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:28.720084 15300 raft_consensus.cc:2454] T e46648198f5343f7a65079887e5df159 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 1.
I20260810 08:00:28.720737 14688 catalog_manager.cc:5587] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601) does not have the latest schema for tablet 5a22a48e2ba843b3a32a8900ab937ea0 (table default.Table_5a6ab9b9acc9408c8811f77acaca9a0c [id=ac3a5787385940bca8030d7bac893a27]). Expected version 2 got 0
I20260810 08:00:28.721346 14918 leader_election.cc:302] T e46648198f5343f7a65079887e5df159 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224; no voters:
I20260810 08:00:28.722386 15640 raft_consensus.cc:2781] T e46648198f5343f7a65079887e5df159 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 08:00:28.722849 15640 raft_consensus.cc:686] T e46648198f5343f7a65079887e5df159 P a91b27762182454baada71e91ceab224 [term 1 LEADER]: Becoming Leader. State: Replica: a91b27762182454baada71e91ceab224, State: Running, Role: LEADER
W20260810 08:00:28.723184 15079 leader_election.cc:341] T 49c35b7f4a72437fa0675abe8a8f3a71 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:28.723631 15079 leader_election.cc:302] T 49c35b7f4a72437fa0675abe8a8f3a71 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:28.723629 15640 consensus_queue.cc:237] T e46648198f5343f7a65079887e5df159 P a91b27762182454baada71e91ceab224 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.724324 15617 raft_consensus.cc:2726] T 49c35b7f4a72437fa0675abe8a8f3a71 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 08:00:28.724792 14680 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet e46648198f5343f7a65079887e5df159 (table default.Table_d81fe7cc77ad45d39c8b44e1299dbefc [id=a705653aa3484a69b369e7e39273d28e], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e46648198f5343f7a65079887e5df159 consensus state has no leader
I20260810 08:00:28.651244 15447 raft_consensus.cc:504] T 35b6da60a5544d7e9b657a5f1e6f90ff P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.726346 15490 tablet_bootstrap.cc:700] T 25de7bf64bf6450ba17b7748568bc0ed P 9d7219686f304f62a54acc7497cbbd07: Time spent opening tablet: real 0.105s user 0.004s sys 0.000s
I20260810 08:00:28.727277 15490 tablet_bootstrap.cc:654] T 25de7bf64bf6450ba17b7748568bc0ed P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:28.729983 14961 tablet_service.cc:1514] Processing DeleteTablet for tablet a1c4f8f514c94128aedd07bd5efdc3be with delete_type TABLET_DATA_DELETED (replaced by tablet a4a78b2a960d4d78aa1f48a9c9edf3be) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:28.731427 15117 tablet_service.cc:1514] Processing DeleteTablet for tablet a1c4f8f514c94128aedd07bd5efdc3be with delete_type TABLET_DATA_DELETED (replaced by tablet a4a78b2a960d4d78aa1f48a9c9edf3be) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:28.732445 15447 leader_election.cc:288] T 35b6da60a5544d7e9b657a5f1e6f90ff P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Requested vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:28.735203 15279 tablet_service.cc:1514] Processing DeleteTablet for tablet a1c4f8f514c94128aedd07bd5efdc3be with delete_type TABLET_DATA_DELETED (replaced by tablet a4a78b2a960d4d78aa1f48a9c9edf3be) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:28.735956 15478 tablet_replica.cc:330] T a1c4f8f514c94128aedd07bd5efdc3be P 3fc5e568f3724f158b616c303efb9eae: stopping tablet replica
I20260810 08:00:28.736706 15478 raft_consensus.cc:2229] T a1c4f8f514c94128aedd07bd5efdc3be P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:28.737157 15478 raft_consensus.cc:2258] T a1c4f8f514c94128aedd07bd5efdc3be P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Raft consensus is shut down!
W20260810 08:00:28.738263 15234 leader_election.cc:341] T 35b6da60a5544d7e9b657a5f1e6f90ff P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Not found: Tablet not found: 35b6da60a5544d7e9b657a5f1e6f90ff
I20260810 08:00:28.738878 15478 ts_tablet_manager.cc:1905] T a1c4f8f514c94128aedd07bd5efdc3be P 3fc5e568f3724f158b616c303efb9eae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:28.738956 15146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b6da60a5544d7e9b657a5f1e6f90ff" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07"
I20260810 08:00:28.739593 15146 raft_consensus.cc:3037] T 35b6da60a5544d7e9b657a5f1e6f90ff P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:28.745412 14961 tablet_service.cc:1467] Processing CreateTablet for tablet e0652a7684b04a89840049ce0d781199 (DEFAULT_TABLE table=default.Table_b7a38b32fda646888162ee796bb97bbf [id=a1fddfa88f634c5480f0282233d121e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:28.746613 14961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0652a7684b04a89840049ce0d781199. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:28.751075 14692 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_5a6ab9b9acc9408c8811f77acaca9a0c" } new_table_name: "default.Table_8aecb3c6298849c38bcafee965161494" modify_external_catalogs: true
I20260810 08:00:28.752836 15439 tablet_bootstrap.cc:654] T 0fae4809ef5e4a9fb16976674e287f03 P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:28.753923 15617 raft_consensus.cc:482] T fbf62f4eab8b49f282df3ab4d95d886f P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:28.754395 15617 raft_consensus.cc:504] T fbf62f4eab8b49f282df3ab4d95d886f P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:28.756392 15617 leader_election.cc:288] T fbf62f4eab8b49f282df3ab4d95d886f P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:28.757546 15302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf62f4eab8b49f282df3ab4d95d886f" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:28.733348 15147 raft_consensus.cc:2454] T e46648198f5343f7a65079887e5df159 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 1.
I20260810 08:00:28.758126 15117 tablet_service.cc:1467] Processing CreateTablet for tablet 8364045e3fa2441e8dc1a1d87f73361d (DEFAULT_TABLE table=default.Table_f3dd0fbbcd664605b91fb032e4877812 [id=97abcf4bf320496ebbec1fd2ed20765b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:28.758457 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf62f4eab8b49f282df3ab4d95d886f" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:28.684629 15479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43002 (request call id 42) took 2401 ms. Trace:
I20260810 08:00:28.759265 15117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8364045e3fa2441e8dc1a1d87f73361d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:28.758999 15479 rpcz_store.cc:276] 0810 08:00:26.282921 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:26.286633 (+ 3712us) service_pool.cc:227] Handling call
0810 08:00:28.469769 (+2183136us) ts_tablet_manager.cc:1145] Deleting tablet 37d8a37a16f34a0285c43c0d71b5f072
0810 08:00:28.469893 (+ 124us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:28.532931 (+ 63038us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:28.645861 (+112930us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:28.684569 (+ 38708us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2182513}
W20260810 08:00:28.760461 15076 leader_election.cc:341] T fbf62f4eab8b49f282df3ab4d95d886f P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:28.760777 15479 tablet_replica.cc:330] T 35b6da60a5544d7e9b657a5f1e6f90ff P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:28.762552 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet 37d8a37a16f34a0285c43c0d71b5f072 (table default.Table_bae5256f457944049d86e384fb3759d6 [id=dad0c6e295df4fdd9b2c27d2b19fbc5f]) successfully deleted
W20260810 08:00:28.763471 15079 leader_election.cc:341] T fbf62f4eab8b49f282df3ab4d95d886f P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:28.763912 15079 leader_election.cc:302] T fbf62f4eab8b49f282df3ab4d95d886f P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:28.767416 15437 tablet_bootstrap.cc:492] T 728f8f80b3bb4b10871d8b400f25377b P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:28.767848 15437 ts_tablet_manager.cc:1397] T 728f8f80b3bb4b10871d8b400f25377b P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.220s user 0.008s sys 0.001s
I20260810 08:00:28.768760 15617 raft_consensus.cc:2726] T fbf62f4eab8b49f282df3ab4d95d886f P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:28.770612 15276 tablet_service.cc:1467] Processing CreateTablet for tablet e0652a7684b04a89840049ce0d781199 (DEFAULT_TABLE table=default.Table_b7a38b32fda646888162ee796bb97bbf [id=a1fddfa88f634c5480f0282233d121e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:28.773499 15279 tablet_service.cc:1467] Processing CreateTablet for tablet 579bac622ddf40bf80d01d66ae2fa100 (DEFAULT_TABLE table=default.Table_64bd369e459e44d9ac85bf7defb3c86c [id=650bc0c97e7842798d575199f1ea662c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:28.776648 14966 tablet_service.cc:1467] Processing CreateTablet for tablet 5990e7ea243843bb869de5dde3e6cd23 (DEFAULT_TABLE table=default.Table_870c8e79c4574433bf1b38ff75564dc9 [id=3def70102e974890b92f64b5c0ff308d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:28.777787 14966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5990e7ea243843bb869de5dde3e6cd23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:28.793032 15490 tablet_bootstrap.cc:492] T 25de7bf64bf6450ba17b7748568bc0ed P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:28.793542 15490 ts_tablet_manager.cc:1397] T 25de7bf64bf6450ba17b7748568bc0ed P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.183s user 0.013s sys 0.000s
I20260810 08:00:28.796159 15490 raft_consensus.cc:348] T 25de7bf64bf6450ba17b7748568bc0ed P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.796818 15490 raft_consensus.cc:374] T 25de7bf64bf6450ba17b7748568bc0ed P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:28.771696 15437 raft_consensus.cc:348] T 728f8f80b3bb4b10871d8b400f25377b P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.797052 15490 raft_consensus.cc:729] T 25de7bf64bf6450ba17b7748568bc0ed P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:28.798636 15490 consensus_queue.cc:260] T 25de7bf64bf6450ba17b7748568bc0ed P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.799798 15437 raft_consensus.cc:374] T 728f8f80b3bb4b10871d8b400f25377b P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:28.800164 15437 raft_consensus.cc:729] T 728f8f80b3bb4b10871d8b400f25377b P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:28.800987 15437 consensus_queue.cc:260] T 728f8f80b3bb4b10871d8b400f25377b P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.804150 15439 tablet_bootstrap.cc:492] T 0fae4809ef5e4a9fb16976674e287f03 P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:28.804558 15439 ts_tablet_manager.cc:1397] T 0fae4809ef5e4a9fb16976674e287f03 P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.137s user 0.007s sys 0.005s
I20260810 08:00:28.808627 14691 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_849976332e234e2bad3d238fc268803f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:28.810163 14691 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_849976332e234e2bad3d238fc268803f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:28.810453 15146 raft_consensus.cc:2454] T 35b6da60a5544d7e9b657a5f1e6f90ff P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 1.
I20260810 08:00:28.810896 15479 raft_consensus.cc:2229] T 35b6da60a5544d7e9b657a5f1e6f90ff P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:28.812978 15479 raft_consensus.cc:2258] T 35b6da60a5544d7e9b657a5f1e6f90ff P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:28.814813 15479 ts_tablet_manager.cc:1905] T 35b6da60a5544d7e9b657a5f1e6f90ff P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:28.819073 15234 leader_election.cc:302] T 35b6da60a5544d7e9b657a5f1e6f90ff P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, 9d7219686f304f62a54acc7497cbbd07; no voters: a91b27762182454baada71e91ceab224
I20260810 08:00:28.821902 15279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 579bac622ddf40bf80d01d66ae2fa100. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:28.822928 15490 ts_tablet_manager.cc:1428] T 25de7bf64bf6450ba17b7748568bc0ed P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
I20260810 08:00:28.823448 15447 raft_consensus.cc:2781] T 35b6da60a5544d7e9b657a5f1e6f90ff P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Leader election won for term 1
I20260810 08:00:28.824433 15276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0652a7684b04a89840049ce0d781199. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:28.826607 15490 tablet_bootstrap.cc:492] T d0761b090d8147b8a5537b0ed51108a9 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:28.828423 14693 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet 924c0f73638f48a88c08d45d459132e1
I20260810 08:00:28.829869 14693 catalog_manager.cc:6565] ReplaceTablet: tablet 924c0f73638f48a88c08d45d459132e1 deleted and replaced by tablet 3281dd757ff44d9498ef70ad372eb150
I20260810 08:00:28.832764 15490 tablet_bootstrap.cc:654] T d0761b090d8147b8a5537b0ed51108a9 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:28.834199 15274 tablet_service.cc:1514] Processing DeleteTablet for tablet 924c0f73638f48a88c08d45d459132e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 3281dd757ff44d9498ef70ad372eb150) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:28.854830 15437 ts_tablet_manager.cc:1428] T 728f8f80b3bb4b10871d8b400f25377b P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.087s user 0.004s sys 0.003s
I20260810 08:00:28.855517 15490 tablet_bootstrap.cc:492] T d0761b090d8147b8a5537b0ed51108a9 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:28.856029 15490 ts_tablet_manager.cc:1397] T d0761b090d8147b8a5537b0ed51108a9 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.004s
I20260810 08:00:28.857242 14693 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_82ef137d777f463c843b60a4dc498741"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:28.858633 15447 raft_consensus.cc:686] T 35b6da60a5544d7e9b657a5f1e6f90ff P 3fc5e568f3724f158b616c303efb9eae [term 1 LEADER]: Becoming Leader. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Running, Role: LEADER
W20260810 08:00:28.859009 14693 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_82ef137d777f463c843b60a4dc498741 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:28.858734 15490 raft_consensus.cc:348] T d0761b090d8147b8a5537b0ed51108a9 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.859434 15490 raft_consensus.cc:374] T d0761b090d8147b8a5537b0ed51108a9 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:28.859972 15490 raft_consensus.cc:729] T d0761b090d8147b8a5537b0ed51108a9 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:28.859601 15447 consensus_queue.cc:237] T 35b6da60a5544d7e9b657a5f1e6f90ff P 3fc5e568f3724f158b616c303efb9eae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.860745 15490 consensus_queue.cc:260] T d0761b090d8147b8a5537b0ed51108a9 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.862563 14960 tablet_service.cc:1514] Processing DeleteTablet for tablet 924c0f73638f48a88c08d45d459132e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 3281dd757ff44d9498ef70ad372eb150) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:28.866237 15439 raft_consensus.cc:348] T 0fae4809ef5e4a9fb16976674e287f03 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.867228 15439 raft_consensus.cc:374] T 0fae4809ef5e4a9fb16976674e287f03 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:28.867511 15439 raft_consensus.cc:729] T 0fae4809ef5e4a9fb16976674e287f03 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:28.871136 15490 ts_tablet_manager.cc:1428] T d0761b090d8147b8a5537b0ed51108a9 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.015s user 0.007s sys 0.001s
I20260810 08:00:28.872115 15490 tablet_bootstrap.cc:492] T 2d295305b5a843eaa84c8ad666355b84 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
W20260810 08:00:28.876732 14690 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8364045e3fa2441e8dc1a1d87f73361d consensus state has no leader
I20260810 08:00:28.877554 15490 tablet_bootstrap.cc:654] T 2d295305b5a843eaa84c8ad666355b84 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:28.887146 15117 tablet_service.cc:1467] Processing CreateTablet for tablet e0652a7684b04a89840049ce0d781199 (DEFAULT_TABLE table=default.Table_b7a38b32fda646888162ee796bb97bbf [id=a1fddfa88f634c5480f0282233d121e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:28.888371 15117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0652a7684b04a89840049ce0d781199. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:28.894079 15439 consensus_queue.cc:260] T 0fae4809ef5e4a9fb16976674e287f03 P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.896502 15439 ts_tablet_manager.cc:1428] T 0fae4809ef5e4a9fb16976674e287f03 P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.092s user 0.006s sys 0.000s
I20260810 08:00:28.899546 15437 tablet_bootstrap.cc:492] T 49c35b7f4a72437fa0675abe8a8f3a71 P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:28.911705 15279 tablet_service.cc:1467] Processing CreateTablet for tablet 5990e7ea243843bb869de5dde3e6cd23 (DEFAULT_TABLE table=default.Table_870c8e79c4574433bf1b38ff75564dc9 [id=3def70102e974890b92f64b5c0ff308d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:28.916178 15280 tablet_service.cc:1467] Processing CreateTablet for tablet e13b91d61063446f9d1454da4f8bb706 (DEFAULT_TABLE table=default.Table_4c0b6e5e875f43e088f4f0215cf39284 [id=b89eea0100b945c1851d323ce8c97591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:28.919278 14690 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_5e2233b19d3d486a86f8dc7190f9e2f1" } modify_external_catalogs: true
I20260810 08:00:28.919862 14690 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_5e2233b19d3d486a86f8dc7190f9e2f1" } modify_external_catalogs: true
I20260810 08:00:28.920267 15478 ts_tablet_manager.cc:1918] T a1c4f8f514c94128aedd07bd5efdc3be P 3fc5e568f3724f158b616c303efb9eae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:28.920620 15478 log.cc:1196] T a1c4f8f514c94128aedd07bd5efdc3be P 3fc5e568f3724f158b616c303efb9eae: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal/wals/a1c4f8f514c94128aedd07bd5efdc3be
I20260810 08:00:28.921284 15478 ts_tablet_manager.cc:1939] T a1c4f8f514c94128aedd07bd5efdc3be P 3fc5e568f3724f158b616c303efb9eae: Deleting consensus metadata
I20260810 08:00:28.923437 15280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e13b91d61063446f9d1454da4f8bb706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:28.929965 15279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5990e7ea243843bb869de5dde3e6cd23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:28.933580 15482 ts_tablet_manager.cc:1918] T 0fae4809ef5e4a9fb16976674e287f03 P a91b27762182454baada71e91ceab224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:28.933997 15482 log.cc:1196] T 0fae4809ef5e4a9fb16976674e287f03 P a91b27762182454baada71e91ceab224: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal/wals/0fae4809ef5e4a9fb16976674e287f03
I20260810 08:00:28.929399 15439 tablet_bootstrap.cc:492] T 49c35b7f4a72437fa0675abe8a8f3a71 P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:28.934829 15482 ts_tablet_manager.cc:1939] T 0fae4809ef5e4a9fb16976674e287f03 P a91b27762182454baada71e91ceab224: Deleting consensus metadata
I20260810 08:00:28.934093 14966 tablet_service.cc:1467] Processing CreateTablet for tablet e13b91d61063446f9d1454da4f8bb706 (DEFAULT_TABLE table=default.Table_4c0b6e5e875f43e088f4f0215cf39284 [id=b89eea0100b945c1851d323ce8c97591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:28.940290 14962 tablet_service.cc:1467] Processing CreateTablet for tablet 579bac622ddf40bf80d01d66ae2fa100 (DEFAULT_TABLE table=default.Table_64bd369e459e44d9ac85bf7defb3c86c [id=650bc0c97e7842798d575199f1ea662c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:28.941903 15640 raft_consensus.cc:482] T 728f8f80b3bb4b10871d8b400f25377b P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:28.942570 15640 raft_consensus.cc:504] T 728f8f80b3bb4b10871d8b400f25377b P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.945748 15447 raft_consensus.cc:482] T 0fae4809ef5e4a9fb16976674e287f03 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:28.946378 15447 raft_consensus.cc:504] T 0fae4809ef5e4a9fb16976674e287f03 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:28.947518 15274 tablet_service.cc:1467] Processing CreateTablet for tablet adada21bc1bf4f19a6347d0935bd74bd (DEFAULT_TABLE table=default.Table_93c814deac714a3288f3980340acbc65 [id=4c70468b9e8649849cd0e79d59be6bb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:28.949009 15274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adada21bc1bf4f19a6347d0935bd74bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:28.949117 15447 leader_election.cc:288] T 0fae4809ef5e4a9fb16976674e287f03 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:28.952884 14681 catalog_manager.cc:4841] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): tablet 0fae4809ef5e4a9fb16976674e287f03 (table default.Table_beb88f8abaa447bfb6962df5718b7313 [id=41bcbdf2ef674d8b8db9284b4b0dcdb9]) successfully deleted
I20260810 08:00:28.953099 15302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "728f8f80b3bb4b10871d8b400f25377b" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
W20260810 08:00:28.955265 14918 leader_election.cc:341] T 728f8f80b3bb4b10871d8b400f25377b P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:28.955062 14960 tablet_service.cc:1467] Processing CreateTablet for tablet adada21bc1bf4f19a6347d0935bd74bd (DEFAULT_TABLE table=default.Table_93c814deac714a3288f3980340acbc65 [id=4c70468b9e8649849cd0e79d59be6bb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:28.957010 15640 leader_election.cc:288] T 728f8f80b3bb4b10871d8b400f25377b P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
W20260810 08:00:28.962976 15234 leader_election.cc:341] T 0fae4809ef5e4a9fb16976674e287f03 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Not found: Tablet not found: 0fae4809ef5e4a9fb16976674e287f03
I20260810 08:00:28.963866 14966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e13b91d61063446f9d1454da4f8bb706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:28.967993 14960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adada21bc1bf4f19a6347d0935bd74bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:28.980089 14962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 579bac622ddf40bf80d01d66ae2fa100. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:28.983718 15439 tablet_bootstrap.cc:654] T 49c35b7f4a72437fa0675abe8a8f3a71 P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:28.985678 14963 tablet_service.cc:1467] Processing CreateTablet for tablet 8e603d04f7454501b2edfb5a1f4ea7f0 (DEFAULT_TABLE table=default.Table_d423114f2dce44ffaf0fad0c40c5b985 [id=1ea3fa32feba46c796a5f7ab529516f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:28.986886 14963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e603d04f7454501b2edfb5a1f4ea7f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:28.986795 15277 tablet_service.cc:1467] Processing CreateTablet for tablet 8e603d04f7454501b2edfb5a1f4ea7f0 (DEFAULT_TABLE table=default.Table_d423114f2dce44ffaf0fad0c40c5b985 [id=1ea3fa32feba46c796a5f7ab529516f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:28.989547 14964 tablet_service.cc:1467] Processing CreateTablet for tablet 389b30cba2b149e9ab0de8c191808db2 (DEFAULT_TABLE table=default.Table_e6b03850596042a6a30ee45ea0720b86 [id=6b0af0270f5d40e1a37b309b130be42a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:28.991065 14964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 389b30cba2b149e9ab0de8c191808db2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:28.996127 15278 tablet_service.cc:1467] Processing CreateTablet for tablet 389b30cba2b149e9ab0de8c191808db2 (DEFAULT_TABLE table=default.Table_e6b03850596042a6a30ee45ea0720b86 [id=6b0af0270f5d40e1a37b309b130be42a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:29.006385 15437 tablet_bootstrap.cc:654] T 49c35b7f4a72437fa0675abe8a8f3a71 P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:29.013237 14965 tablet_service.cc:1467] Processing CreateTablet for tablet 3ac554bd37b045a2a8cf2743f4645f28 (DEFAULT_TABLE table=default.Table_e4e38aa2c28146cabb97071119b73368 [id=e356df0e1219419c9eaab96530f49ed7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:29.014271 14965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ac554bd37b045a2a8cf2743f4645f28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:29.020216 15439 tablet_bootstrap.cc:492] T 49c35b7f4a72437fa0675abe8a8f3a71 P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:29.020799 15439 ts_tablet_manager.cc:1397] T 49c35b7f4a72437fa0675abe8a8f3a71 P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.092s user 0.008s sys 0.004s
I20260810 08:00:29.027745 15482 tablet_replica.cc:330] T a1c4f8f514c94128aedd07bd5efdc3be P a91b27762182454baada71e91ceab224: stopping tablet replica
I20260810 08:00:29.028716 15482 raft_consensus.cc:2229] T a1c4f8f514c94128aedd07bd5efdc3be P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:29.029381 15482 raft_consensus.cc:2258] T a1c4f8f514c94128aedd07bd5efdc3be P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:29.030426 15277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e603d04f7454501b2edfb5a1f4ea7f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:29.031127 15482 ts_tablet_manager.cc:1905] T a1c4f8f514c94128aedd07bd5efdc3be P a91b27762182454baada71e91ceab224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:29.036181 14678 catalog_manager.cc:4841] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): tablet a1c4f8f514c94128aedd07bd5efdc3be (table default.Table_89fe7ad723964551909b73ff6b72b9af [id=ce96e7fb7c46404a859ca08c1fd1b26a]) successfully deleted
I20260810 08:00:29.036777 15278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 389b30cba2b149e9ab0de8c191808db2. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 08:00:29.037017 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet 924c0f73638f48a88c08d45d459132e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 924c0f73638f48a88c08d45d459132e1 already in progress: creating tablet
I20260810 08:00:29.039410 15447 raft_consensus.cc:482] T 114d3eac994f466bb1787e066de05e2c P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:29.039877 15447 raft_consensus.cc:504] T 114d3eac994f466bb1787e066de05e2c P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:29.041837 15447 leader_election.cc:288] T 114d3eac994f466bb1787e066de05e2c P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:29.042634 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "114d3eac994f466bb1787e066de05e2c" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:29.043270 14987 raft_consensus.cc:2454] T 114d3eac994f466bb1787e066de05e2c P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 0.
I20260810 08:00:29.044854 15234 leader_election.cc:302] T 114d3eac994f466bb1787e066de05e2c P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224; no voters:
I20260810 08:00:29.046103 15447 raft_consensus.cc:2781] T 114d3eac994f466bb1787e066de05e2c P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:29.046430 15447 raft_consensus.cc:482] T 114d3eac994f466bb1787e066de05e2c P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:29.046730 15447 raft_consensus.cc:3037] T 114d3eac994f466bb1787e066de05e2c P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:29.032263 15439 raft_consensus.cc:348] T 49c35b7f4a72437fa0675abe8a8f3a71 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:29.047297 15439 raft_consensus.cc:374] T 49c35b7f4a72437fa0675abe8a8f3a71 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:29.047536 15439 raft_consensus.cc:729] T 49c35b7f4a72437fa0675abe8a8f3a71 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:29.054710 15439 consensus_queue.cc:260] T 49c35b7f4a72437fa0675abe8a8f3a71 P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:29.056962 15662 raft_consensus.cc:482] T 49c35b7f4a72437fa0675abe8a8f3a71 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:29.057361 15662 raft_consensus.cc:504] T 49c35b7f4a72437fa0675abe8a8f3a71 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:29.060743 15662 leader_election.cc:288] T 49c35b7f4a72437fa0675abe8a8f3a71 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:29.063032 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49c35b7f4a72437fa0675abe8a8f3a71" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
W20260810 08:00:29.064476 15234 leader_election.cc:341] T 49c35b7f4a72437fa0675abe8a8f3a71 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:29.068944 14695 catalog_manager.cc:5587] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601) does not have the latest schema for tablet 5a22a48e2ba843b3a32a8900ab937ea0 (table default.Table_5a6ab9b9acc9408c8811f77acaca9a0c [id=ac3a5787385940bca8030d7bac893a27]). Expected version 2 got 0
I20260810 08:00:29.070266 14695 catalog_manager.cc:5495] T e46648198f5343f7a65079887e5df159 P a91b27762182454baada71e91ceab224 reported cstate change: term changed from 0 to 1, leader changed from <none> to a91b27762182454baada71e91ceab224 (127.14.57.193). New cstate: current_term: 1 leader_uuid: "a91b27762182454baada71e91ceab224" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } health_report { overall_health: UNKNOWN } } }
I20260810 08:00:29.072311 14695 catalog_manager.cc:5587] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601) does not have the latest schema for tablet e46648198f5343f7a65079887e5df159 (table default.Table_d81fe7cc77ad45d39c8b44e1299dbefc [id=a705653aa3484a69b369e7e39273d28e]). Expected version 1 got 0
I20260810 08:00:29.073499 14686 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet 1461cdeda5664afd9c4c2b45199179b6
I20260810 08:00:29.074455 15439 ts_tablet_manager.cc:1428] T 49c35b7f4a72437fa0675abe8a8f3a71 P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.053s user 0.006s sys 0.002s
I20260810 08:00:29.078289 15275 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ddae63b29c34e1397c5765e52e1478f with delete_type TABLET_DATA_DELETED (replaced by tablet 561cbc6a4eb14d3584f88ea26a0fea7a) from {username='slave'} at 127.0.0.1:40692
W20260810 08:00:29.078476 14689 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.14.57.195:51883 (request call id 36) took 1882 ms (client timeout 1000 ms). Trace:
I20260810 08:00:29.078797 15273 tablet_service.cc:1514] Processing DeleteTablet for tablet 2faab63c5023431c98e7b01ce61e04c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:24 UTC) from {username='slave'} at 127.0.0.1:40692
W20260810 08:00:29.078821 14689 rpcz_store.cc:269] 0810 08:00:27.195900 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:27.196011 (+ 111us) service_pool.cc:227] Handling call
0810 08:00:27.201803 (+ 5792us) spinlock_profiling.cc:243] Waited 4.61 ms on lock 0x7b5400021554. stack: 00007fe2372c4033 00007fe2372c3f30 00007fe236e45534 00000000004db2b4 00000000004db249 00007fe23e1d0af0 00007fe23e253f2a 00007fe24011fd6c 00007fe240219c3e 00007fe23dd50fb4 00007fe23dd50f42 00007fe23dd50e71 00007fe23dd50dec 00007fe23dd500a2 00007fe23bffa94c 00007fe23bff9d86
0810 08:00:29.078389 (+1876586us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1,"spinlock_wait_cycles":10613504}
I20260810 08:00:29.079563 15272 tablet_service.cc:1514] Processing DeleteTablet for tablet 35b6da60a5544d7e9b657a5f1e6f90ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:26 UTC) from {username='slave'} at 127.0.0.1:40692
W20260810 08:00:29.079586 14687 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.14.57.194:55587 (request call id 41) took 921 ms (client timeout 1000 ms). Trace:
I20260810 08:00:29.079867 14689 master_service.cc:946] ReplaceTablet: received request to replace tablet 8364045e3fa2441e8dc1a1d87f73361d from {username='slave'} at 127.0.0.1:42300
W20260810 08:00:29.079913 14687 rpcz_store.cc:269] 0810 08:00:28.158337 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:28.158487 (+ 150us) service_pool.cc:227] Handling call
0810 08:00:28.165146 (+ 6659us) spinlock_profiling.cc:243] Waited 4.6 ms on lock 0x7b5400021554. stack: 00007fe2372c4033 00007fe2372c3f30 00007fe236e45534 00000000004db2b4 00000000004db249 00007fe23e1d0af0 00007fe23e253f2a 00007fe24011fd6c 00007fe240219c3e 00007fe23dd50fb4 00007fe23dd50f42 00007fe23dd50e71 00007fe23dd50dec 00007fe23dd500a2 00007fe23bffa94c 00007fe23bff9d86
0810 08:00:29.079528 (+914382us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1,"spinlock_wait_cycles":10572672}
I20260810 08:00:29.080415 15276 tablet_service.cc:1514] Processing DeleteTablet for tablet 1461cdeda5664afd9c4c2b45199179b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:28 UTC) from {username='slave'} at 127.0.0.1:40692
W20260810 08:00:29.080611 14694 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.14.57.195:51883 (request call id 37) took 880 ms (client timeout 1000 ms). Trace:
W20260810 08:00:29.080907 14694 rpcz_store.cc:269] 0810 08:00:28.200662 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:28.200788 (+ 126us) service_pool.cc:227] Handling call
0810 08:00:29.080553 (+879765us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":4,"spinlock_wait_cycles":2304}
I20260810 08:00:29.083199 15478 tablet_replica.cc:330] T 9ddae63b29c34e1397c5765e52e1478f P 3fc5e568f3724f158b616c303efb9eae: stopping tablet replica
I20260810 08:00:29.083911 15478 raft_consensus.cc:2229] T 9ddae63b29c34e1397c5765e52e1478f P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:29.084363 15478 pending_rounds.cc:62] T 9ddae63b29c34e1397c5765e52e1478f P 3fc5e568f3724f158b616c303efb9eae: Trying to abort 1 pending ops.
I20260810 08:00:29.084612 15478 pending_rounds.cc:68] T 9ddae63b29c34e1397c5765e52e1478f P 3fc5e568f3724f158b616c303efb9eae: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316884792198668288 op_type: NO_OP noop_request { }
I20260810 08:00:29.084981 15478 raft_consensus.cc:2866] T 9ddae63b29c34e1397c5765e52e1478f P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260810 08:00:29.085312 15478 raft_consensus.cc:2258] T 9ddae63b29c34e1397c5765e52e1478f P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:29.089015 15478 ts_tablet_manager.cc:1905] T 9ddae63b29c34e1397c5765e52e1478f P 3fc5e568f3724f158b616c303efb9eae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:29.090061 15272 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8454f693494ab3b82f5b85c2d62bb2 (DEFAULT_TABLE table=default.Table_8ac532f4969e45a197db311de2a7b795 [id=7893824f2c374508b7b3c4635ebea5b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:29.091491 15272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8454f693494ab3b82f5b85c2d62bb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:29.097443 15276 tablet_service.cc:1467] Processing CreateTablet for tablet 3ac554bd37b045a2a8cf2743f4645f28 (DEFAULT_TABLE table=default.Table_e4e38aa2c28146cabb97071119b73368 [id=e356df0e1219419c9eaab96530f49ed7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:29.100893 15439 tablet_bootstrap.cc:492] T 728f8f80b3bb4b10871d8b400f25377b P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:29.106797 15126 tablet_service.cc:1514] Processing DeleteTablet for tablet 924c0f73638f48a88c08d45d459132e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 3281dd757ff44d9498ef70ad372eb150) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:29.107384 15439 tablet_bootstrap.cc:654] T 728f8f80b3bb4b10871d8b400f25377b P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:29.108701 15146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fae4809ef5e4a9fb16976674e287f03" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:29.109139 15125 tablet_service.cc:1467] Processing CreateTablet for tablet 579bac622ddf40bf80d01d66ae2fa100 (DEFAULT_TABLE table=default.Table_64bd369e459e44d9ac85bf7defb3c86c [id=650bc0c97e7842798d575199f1ea662c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:29.109357 15146 raft_consensus.cc:2454] T 0fae4809ef5e4a9fb16976674e287f03 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 0.
I20260810 08:00:29.110450 15125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 579bac622ddf40bf80d01d66ae2fa100. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:29.111032 15147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49c35b7f4a72437fa0675abe8a8f3a71" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:29.111608 15147 raft_consensus.cc:2454] T 49c35b7f4a72437fa0675abe8a8f3a71 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 0.
I20260810 08:00:29.112804 15234 leader_election.cc:302] T 0fae4809ef5e4a9fb16976674e287f03 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, 9d7219686f304f62a54acc7497cbbd07; no voters: a91b27762182454baada71e91ceab224
W20260810 08:00:29.113915 15234 leader_election.cc:341] T 114d3eac994f466bb1787e066de05e2c P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Not found: Tablet not found: 114d3eac994f466bb1787e066de05e2c
I20260810 08:00:29.114645 15234 leader_election.cc:302] T 49c35b7f4a72437fa0675abe8a8f3a71 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, 9d7219686f304f62a54acc7497cbbd07; no voters: a91b27762182454baada71e91ceab224
I20260810 08:00:29.115454 15662 raft_consensus.cc:2781] T 0fae4809ef5e4a9fb16976674e287f03 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:29.115792 15662 raft_consensus.cc:482] T 0fae4809ef5e4a9fb16976674e287f03 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:29.116089 15662 raft_consensus.cc:3037] T 0fae4809ef5e4a9fb16976674e287f03 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:29.123086 15670 raft_consensus.cc:2781] T 49c35b7f4a72437fa0675abe8a8f3a71 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:29.123613 15670 raft_consensus.cc:482] T 49c35b7f4a72437fa0675abe8a8f3a71 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:29.124068 15670 raft_consensus.cc:3037] T 49c35b7f4a72437fa0675abe8a8f3a71 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:29.126333 15447 raft_consensus.cc:504] T 114d3eac994f466bb1787e066de05e2c P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:29.127697 15118 tablet_service.cc:1467] Processing CreateTablet for tablet e13b91d61063446f9d1454da4f8bb706 (DEFAULT_TABLE table=default.Table_4c0b6e5e875f43e088f4f0215cf39284 [id=b89eea0100b945c1851d323ce8c97591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:29.127975 15119 tablet_service.cc:1467] Processing CreateTablet for tablet 8e603d04f7454501b2edfb5a1f4ea7f0 (DEFAULT_TABLE table=default.Table_d423114f2dce44ffaf0fad0c40c5b985 [id=1ea3fa32feba46c796a5f7ab529516f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:29.128422 15447 leader_election.cc:288] T 114d3eac994f466bb1787e066de05e2c P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Requested vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:29.128880 15118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e13b91d61063446f9d1454da4f8bb706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:29.129518 15122 tablet_service.cc:1467] Processing CreateTablet for tablet 389b30cba2b149e9ab0de8c191808db2 (DEFAULT_TABLE table=default.Table_e6b03850596042a6a30ee45ea0720b86 [id=6b0af0270f5d40e1a37b309b130be42a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:29.129349 15126 tablet_service.cc:1467] Processing CreateTablet for tablet 5990e7ea243843bb869de5dde3e6cd23 (DEFAULT_TABLE table=default.Table_870c8e79c4574433bf1b38ff75564dc9 [id=3def70102e974890b92f64b5c0ff308d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:29.130332 15120 tablet_service.cc:1467] Processing CreateTablet for tablet adada21bc1bf4f19a6347d0935bd74bd (DEFAULT_TABLE table=default.Table_93c814deac714a3288f3980340acbc65 [id=4c70468b9e8649849cd0e79d59be6bb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:29.130613 15122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 389b30cba2b149e9ab0de8c191808db2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:29.131412 15120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adada21bc1bf4f19a6347d0935bd74bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:29.132612 15147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "728f8f80b3bb4b10871d8b400f25377b" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:29.133215 15147 raft_consensus.cc:2454] T 728f8f80b3bb4b10871d8b400f25377b P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 0.
I20260810 08:00:29.134090 15126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5990e7ea243843bb869de5dde3e6cd23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:29.135841 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "114d3eac994f466bb1787e066de05e2c" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224"
I20260810 08:00:29.136518 14987 raft_consensus.cc:3037] T 114d3eac994f466bb1787e066de05e2c P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:29.147246 15119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e603d04f7454501b2edfb5a1f4ea7f0. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 08:00:29.148442 15234 leader_election.cc:341] T 114d3eac994f466bb1787e066de05e2c P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Not found: Tablet not found: 114d3eac994f466bb1787e066de05e2c
I20260810 08:00:29.158449 14922 leader_election.cc:302] T 728f8f80b3bb4b10871d8b400f25377b P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224; no voters: 3fc5e568f3724f158b616c303efb9eae
I20260810 08:00:29.159842 15640 raft_consensus.cc:2781] T 728f8f80b3bb4b10871d8b400f25377b P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:29.160187 15640 raft_consensus.cc:482] T 728f8f80b3bb4b10871d8b400f25377b P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:29.160552 15640 raft_consensus.cc:3037] T 728f8f80b3bb4b10871d8b400f25377b P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
W20260810 08:00:29.168934 15076 consensus_peers.cc:487] T a1c4f8f514c94128aedd07bd5efdc3be P 9d7219686f304f62a54acc7497cbbd07 -> Peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Couldn't send request to peer 3fc5e568f3724f158b616c303efb9eae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a1c4f8f514c94128aedd07bd5efdc3be. This is attempt 1: this message will repeat every 5th retry.
I20260810 08:00:29.172250 15623 raft_consensus.cc:482] T 25de7bf64bf6450ba17b7748568bc0ed P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:29.172719 15623 raft_consensus.cc:504] T 25de7bf64bf6450ba17b7748568bc0ed P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:29.173055 15617 consensus_queue.cc:1035] T 5a22a48e2ba843b3a32a8900ab937ea0 P 9d7219686f304f62a54acc7497cbbd07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 08:00:29.174626 15623 leader_election.cc:288] T 25de7bf64bf6450ba17b7748568bc0ed P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), a91b27762182454baada71e91ceab224 (127.14.57.193:43601)
I20260810 08:00:29.177188 15437 tablet_bootstrap.cc:492] T 49c35b7f4a72437fa0675abe8a8f3a71 P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:29.181728 14989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25de7bf64bf6450ba17b7748568bc0ed" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:29.183593 15437 ts_tablet_manager.cc:1397] T 49c35b7f4a72437fa0675abe8a8f3a71 P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.287s user 0.008s sys 0.005s
I20260810 08:00:29.183840 15490 tablet_bootstrap.cc:492] T 2d295305b5a843eaa84c8ad666355b84 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:29.184268 15490 ts_tablet_manager.cc:1397] T 2d295305b5a843eaa84c8ad666355b84 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.312s user 0.031s sys 0.095s
W20260810 08:00:29.183585 15079 leader_election.cc:341] T 25de7bf64bf6450ba17b7748568bc0ed P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:29.186450 15302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25de7bf64bf6450ba17b7748568bc0ed" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
W20260810 08:00:29.197024 15076 leader_election.cc:341] T 25de7bf64bf6450ba17b7748568bc0ed P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:29.197503 15076 leader_election.cc:302] T 25de7bf64bf6450ba17b7748568bc0ed P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:29.198406 15623 raft_consensus.cc:2726] T 25de7bf64bf6450ba17b7748568bc0ed P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:29.198028 15437 raft_consensus.cc:348] T 49c35b7f4a72437fa0675abe8a8f3a71 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:29.198848 15437 raft_consensus.cc:374] T 49c35b7f4a72437fa0675abe8a8f3a71 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:29.200498 15437 raft_consensus.cc:729] T 49c35b7f4a72437fa0675abe8a8f3a71 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:29.204226 15437 consensus_queue.cc:260] T 49c35b7f4a72437fa0675abe8a8f3a71 P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:29.216018 15437 ts_tablet_manager.cc:1428] T 49c35b7f4a72437fa0675abe8a8f3a71 P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.032s user 0.004s sys 0.003s
I20260810 08:00:29.221191 15437 tablet_bootstrap.cc:492] T 37d8a37a16f34a0285c43c0d71b5f072 P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:29.231526 15276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ac554bd37b045a2a8cf2743f4645f28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:29.235888 14686 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_d612355f9d634ad88ba3d45cf5ace9ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:29.238095 15662 raft_consensus.cc:504] T 0fae4809ef5e4a9fb16976674e287f03 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:29.240121 15662 leader_election.cc:288] T 0fae4809ef5e4a9fb16976674e287f03 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Requested vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
W20260810 08:00:29.241967 14686 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d612355f9d634ad88ba3d45cf5ace9ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:29.242085 14987 raft_consensus.cc:2454] T 114d3eac994f466bb1787e066de05e2c P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 1.
I20260810 08:00:29.243839 15234 leader_election.cc:302] T 114d3eac994f466bb1787e066de05e2c P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224; no voters: 9d7219686f304f62a54acc7497cbbd07
W20260810 08:00:29.246707 15234 leader_election.cc:341] T 0fae4809ef5e4a9fb16976674e287f03 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Not found: Tablet not found: 0fae4809ef5e4a9fb16976674e287f03
I20260810 08:00:29.247635 15662 raft_consensus.cc:2781] T 114d3eac994f466bb1787e066de05e2c P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Leader election won for term 1
I20260810 08:00:29.248152 15662 raft_consensus.cc:686] T 114d3eac994f466bb1787e066de05e2c P 3fc5e568f3724f158b616c303efb9eae [term 1 LEADER]: Becoming Leader. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Running, Role: LEADER
I20260810 08:00:29.248958 15662 consensus_queue.cc:237] T 114d3eac994f466bb1787e066de05e2c P 3fc5e568f3724f158b616c303efb9eae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:29.252007 15147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fae4809ef5e4a9fb16976674e287f03" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07"
I20260810 08:00:29.252671 15147 raft_consensus.cc:3037] T 0fae4809ef5e4a9fb16976674e287f03 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:29.254180 15490 raft_consensus.cc:348] T 2d295305b5a843eaa84c8ad666355b84 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:29.255451 15490 raft_consensus.cc:374] T 2d295305b5a843eaa84c8ad666355b84 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:29.257153 15490 raft_consensus.cc:729] T 2d295305b5a843eaa84c8ad666355b84 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:29.259435 15490 consensus_queue.cc:260] T 2d295305b5a843eaa84c8ad666355b84 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:29.270722 15123 tablet_service.cc:1467] Processing CreateTablet for tablet 3ac554bd37b045a2a8cf2743f4645f28 (DEFAULT_TABLE table=default.Table_e4e38aa2c28146cabb97071119b73368 [id=e356df0e1219419c9eaab96530f49ed7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:29.271863 15121 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8454f693494ab3b82f5b85c2d62bb2 (DEFAULT_TABLE table=default.Table_8ac532f4969e45a197db311de2a7b795 [id=7893824f2c374508b7b3c4635ebea5b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:29.272799 15479 ts_tablet_manager.cc:1918] T 35b6da60a5544d7e9b657a5f1e6f90ff P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:29.273140 15479 log.cc:1196] T 35b6da60a5544d7e9b657a5f1e6f90ff P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/35b6da60a5544d7e9b657a5f1e6f90ff
I20260810 08:00:29.271855 15123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ac554bd37b045a2a8cf2743f4645f28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:29.282166 15121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8454f693494ab3b82f5b85c2d62bb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:29.282176 15479 ts_tablet_manager.cc:1939] T 35b6da60a5544d7e9b657a5f1e6f90ff P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:29.321596 15670 raft_consensus.cc:504] T 49c35b7f4a72437fa0675abe8a8f3a71 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:29.323734 15670 leader_election.cc:288] T 49c35b7f4a72437fa0675abe8a8f3a71 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Requested vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:29.324577 15147 raft_consensus.cc:2454] T 0fae4809ef5e4a9fb16976674e287f03 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 1.
I20260810 08:00:29.331200 15234 leader_election.cc:302] T 0fae4809ef5e4a9fb16976674e287f03 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, 9d7219686f304f62a54acc7497cbbd07; no voters: a91b27762182454baada71e91ceab224
I20260810 08:00:29.333341 15147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49c35b7f4a72437fa0675abe8a8f3a71" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07"
I20260810 08:00:29.334012 15147 raft_consensus.cc:3037] T 49c35b7f4a72437fa0675abe8a8f3a71 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:29.335342 15117 tablet_service.cc:1514] Processing DeleteTablet for tablet 1461cdeda5664afd9c4c2b45199179b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:28 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:29.337808 15670 raft_consensus.cc:2781] T 0fae4809ef5e4a9fb16976674e287f03 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Leader election won for term 1
I20260810 08:00:29.338976 15670 raft_consensus.cc:686] T 0fae4809ef5e4a9fb16976674e287f03 P 3fc5e568f3724f158b616c303efb9eae [term 1 LEADER]: Becoming Leader. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Running, Role: LEADER
I20260810 08:00:29.339736 15670 consensus_queue.cc:237] T 0fae4809ef5e4a9fb16976674e287f03 P 3fc5e568f3724f158b616c303efb9eae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:29.344478 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49c35b7f4a72437fa0675abe8a8f3a71" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224"
I20260810 08:00:29.344282 15640 raft_consensus.cc:504] T 728f8f80b3bb4b10871d8b400f25377b P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:29.345017 14987 raft_consensus.cc:3037] T 49c35b7f4a72437fa0675abe8a8f3a71 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:29.350395 15146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "728f8f80b3bb4b10871d8b400f25377b" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07"
I20260810 08:00:29.350982 15146 raft_consensus.cc:3037] T 728f8f80b3bb4b10871d8b400f25377b P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:29.355495 14966 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8454f693494ab3b82f5b85c2d62bb2 (DEFAULT_TABLE table=default.Table_8ac532f4969e45a197db311de2a7b795 [id=7893824f2c374508b7b3c4635ebea5b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:29.356599 14966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8454f693494ab3b82f5b85c2d62bb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:29.359452 15617 raft_consensus.cc:482] T d0761b090d8147b8a5537b0ed51108a9 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:29.360080 15437 tablet_bootstrap.cc:654] T 37d8a37a16f34a0285c43c0d71b5f072 P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:29.359939 15617 raft_consensus.cc:504] T d0761b090d8147b8a5537b0ed51108a9 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:29.363343 15302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0761b090d8147b8a5537b0ed51108a9" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:29.364004 15617 leader_election.cc:288] T d0761b090d8147b8a5537b0ed51108a9 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), a91b27762182454baada71e91ceab224 (127.14.57.193:43601)
I20260810 08:00:29.365134 15302 raft_consensus.cc:3037] T 5a22a48e2ba843b3a32a8900ab937ea0 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:29.365686 15490 ts_tablet_manager.cc:1428] T 2d295305b5a843eaa84c8ad666355b84 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.181s user 0.007s sys 0.001s
I20260810 08:00:29.366582 15490 tablet_bootstrap.cc:492] T d22a778bfe784b2f8a7b4d2216909f10 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
W20260810 08:00:29.368095 15076 leader_election.cc:341] T d0761b090d8147b8a5537b0ed51108a9 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
W20260810 08:00:29.368409 14681 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.57.252:33819, user_credentials={real_user=slave}} blocked reactor thread for 97226.4us
I20260810 08:00:29.374863 15479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43002 (request call id 47) took 2734 ms. Trace:
I20260810 08:00:29.375201 15479 rpcz_store.cc:276] 0810 08:00:26.640013 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:26.640165 (+ 152us) service_pool.cc:227] Handling call
0810 08:00:28.760590 (+2120425us) ts_tablet_manager.cc:1145] Deleting tablet 35b6da60a5544d7e9b657a5f1e6f90ff
0810 08:00:28.760659 (+ 69us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:28.812742 (+ 52083us) spinlock_profiling.cc:243] Waited 49.4 ms on lock 0x7b54000b2a54. stack: 00007f8ef5494033 00007f8ef5493f30 00007f8ef5015534 00000000004db2b4 00000000004db249 00007f8efc3a0471 00007f8efc423352 00007f8efc83926c 00007f8efd446cfa 00007f8efd476e10 00007f8efd45de0d 00007f8efd45ddb9 00007f8efd45dd49 00007f8efd45dd11 00007f8efd45cf3d 0000000000580784
0810 08:00:28.925869 (+113127us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:29.272581 (+346712us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:29.374786 (+102205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":113624448,"tablet-delete.queue_time_us":2119856}
I20260810 08:00:29.376276 15479 tablet_replica.cc:330] T 0c152aab7e244afd857534869b13815c P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:29.376961 15479 raft_consensus.cc:2229] T 0c152aab7e244afd857534869b13815c P 9d7219686f304f62a54acc7497cbbd07 [term 1 LEADER]: Raft consensus shutting down.
I20260810 08:00:29.381163 14691 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_9d3325c7496340d2bf7e36c9618fd581" } new_table_name: "default.Table_290c3db6ca0e4fdf8e007f723b11d27d" modify_external_catalogs: true
I20260810 08:00:29.385155 14687 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_bfb2fd6d4f974132b165f34aedb323bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:29.386900 14687 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bfb2fd6d4f974132b165f34aedb323bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:29.417364 15439 tablet_bootstrap.cc:492] T 728f8f80b3bb4b10871d8b400f25377b P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:29.417800 15439 ts_tablet_manager.cc:1397] T 728f8f80b3bb4b10871d8b400f25377b P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.317s user 0.033s sys 0.025s
I20260810 08:00:29.417129 15479 raft_consensus.cc:2258] T 0c152aab7e244afd857534869b13815c P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:29.419956 15439 raft_consensus.cc:348] T 728f8f80b3bb4b10871d8b400f25377b P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:29.420629 15439 raft_consensus.cc:374] T 728f8f80b3bb4b10871d8b400f25377b P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:29.420878 15439 raft_consensus.cc:729] T 728f8f80b3bb4b10871d8b400f25377b P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:29.430810 15662 raft_consensus.cc:482] T 5a22a48e2ba843b3a32a8900ab937ea0 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:29.431321 15662 raft_consensus.cc:504] T 5a22a48e2ba843b3a32a8900ab937ea0 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:29.434990 15617 consensus_queue.cc:1035] T 5a22a48e2ba843b3a32a8900ab937ea0 P 9d7219686f304f62a54acc7497cbbd07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 08:00:29.437076 15439 consensus_queue.cc:260] T 728f8f80b3bb4b10871d8b400f25377b P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:29.439607 15439 ts_tablet_manager.cc:1428] T 728f8f80b3bb4b10871d8b400f25377b P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.022s user 0.005s sys 0.000s
I20260810 08:00:29.440234 15662 leader_election.cc:288] T 5a22a48e2ba843b3a32a8900ab937ea0 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:29.440995 14690 catalog_manager.cc:5777] Not sending DeleteTablet requests; no consensus state for tablet 47ce047737e7461c9bfc62ce8491293e
I20260810 08:00:29.443156 15146 raft_consensus.cc:2454] T 728f8f80b3bb4b10871d8b400f25377b P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 1.
I20260810 08:00:29.448107 15146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a22a48e2ba843b3a32a8900ab937ea0" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:29.451334 15439 tablet_bootstrap.cc:492] T 37d8a37a16f34a0285c43c0d71b5f072 P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:29.453280 14963 tablet_service.cc:1514] Processing DeleteTablet for tablet 1461cdeda5664afd9c4c2b45199179b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:28 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:29.346362 15640 leader_election.cc:288] T 728f8f80b3bb4b10871d8b400f25377b P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Requested vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:29.457257 14989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a22a48e2ba843b3a32a8900ab937ea0" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:29.458937 14985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0761b090d8147b8a5537b0ed51108a9" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:29.469672 15300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "728f8f80b3bb4b10871d8b400f25377b" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae"
I20260810 08:00:29.470561 15300 raft_consensus.cc:3037] T 728f8f80b3bb4b10871d8b400f25377b P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:29.472921 15234 leader_election.cc:302] T 5a22a48e2ba843b3a32a8900ab937ea0 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae; no voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224
I20260810 08:00:29.474138 15670 raft_consensus.cc:2726] T 5a22a48e2ba843b3a32a8900ab937ea0 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 08:00:29.479938 15631 consensus_queue.cc:1035] T 2faab63c5023431c98e7b01ce61e04c2 P a91b27762182454baada71e91ceab224 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 08:00:29.487846 15439 tablet_bootstrap.cc:654] T 37d8a37a16f34a0285c43c0d71b5f072 P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
W20260810 08:00:29.490931 15079 leader_election.cc:341] T d0761b090d8147b8a5537b0ed51108a9 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:29.491384 15079 leader_election.cc:302] T d0761b090d8147b8a5537b0ed51108a9 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:29.492493 15623 raft_consensus.cc:2726] T d0761b090d8147b8a5537b0ed51108a9 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 08:00:29.492596 15079 consensus_peers.cc:487] T a1c4f8f514c94128aedd07bd5efdc3be P 9d7219686f304f62a54acc7497cbbd07 -> Peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Couldn't send request to peer a91b27762182454baada71e91ceab224. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260810 08:00:29.512733 15490 tablet_bootstrap.cc:654] T d22a778bfe784b2f8a7b4d2216909f10 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:29.526640 14922 leader_election.cc:302] T 728f8f80b3bb4b10871d8b400f25377b P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224; no voters:
I20260810 08:00:29.527503 15640 raft_consensus.cc:2781] T 728f8f80b3bb4b10871d8b400f25377b P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 08:00:29.527933 15640 raft_consensus.cc:686] T 728f8f80b3bb4b10871d8b400f25377b P a91b27762182454baada71e91ceab224 [term 1 LEADER]: Becoming Leader. State: Replica: a91b27762182454baada71e91ceab224, State: Running, Role: LEADER
I20260810 08:00:29.528692 15640 consensus_queue.cc:237] T 728f8f80b3bb4b10871d8b400f25377b P a91b27762182454baada71e91ceab224 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:29.533748 15439 tablet_bootstrap.cc:492] T 37d8a37a16f34a0285c43c0d71b5f072 P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:29.545868 15479 ts_tablet_manager.cc:1905] T 0c152aab7e244afd857534869b13815c P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 08:00:29.549072 14922 consensus_peers.cc:487] T 2faab63c5023431c98e7b01ce61e04c2 P a91b27762182454baada71e91ceab224 -> Peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Couldn't send request to peer 9d7219686f304f62a54acc7497cbbd07. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2faab63c5023431c98e7b01ce61e04c2. This is attempt 1: this message will repeat every 5th retry.
I20260810 08:00:29.553990 15439 ts_tablet_manager.cc:1397] T 37d8a37a16f34a0285c43c0d71b5f072 P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.103s user 0.008s sys 0.008s
I20260810 08:00:29.557107 15439 raft_consensus.cc:348] T 37d8a37a16f34a0285c43c0d71b5f072 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:29.570601 15439 raft_consensus.cc:374] T 37d8a37a16f34a0285c43c0d71b5f072 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:29.570884 15439 raft_consensus.cc:729] T 37d8a37a16f34a0285c43c0d71b5f072 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:29.571617 15439 consensus_queue.cc:260] T 37d8a37a16f34a0285c43c0d71b5f072 P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:29.579603 14693 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_e4e38aa2c28146cabb97071119b73368" } modify_external_catalogs: true
I20260810 08:00:29.582041 15300 raft_consensus.cc:2454] T 728f8f80b3bb4b10871d8b400f25377b P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 1.
I20260810 08:00:29.583841 14987 raft_consensus.cc:2454] T 49c35b7f4a72437fa0675abe8a8f3a71 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 1.
I20260810 08:00:29.585060 15147 raft_consensus.cc:2454] T 49c35b7f4a72437fa0675abe8a8f3a71 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 1.
W20260810 08:00:29.586683 14960 consensus_meta.cc:287] T adada21bc1bf4f19a6347d0935bd74bd P a91b27762182454baada71e91ceab224: Time spent flushing consensus metadata: real 0.582s user 0.010s sys 0.000s
I20260810 08:00:29.581281 14693 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_e4e38aa2c28146cabb97071119b73368" } modify_external_catalogs: true
I20260810 08:00:29.611509 15482 ts_tablet_manager.cc:1918] T a1c4f8f514c94128aedd07bd5efdc3be P a91b27762182454baada71e91ceab224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:29.611927 15482 log.cc:1196] T a1c4f8f514c94128aedd07bd5efdc3be P a91b27762182454baada71e91ceab224: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal/wals/a1c4f8f514c94128aedd07bd5efdc3be
I20260810 08:00:29.612581 15482 ts_tablet_manager.cc:1939] T a1c4f8f514c94128aedd07bd5efdc3be P a91b27762182454baada71e91ceab224: Deleting consensus metadata
I20260810 08:00:29.629465 14689 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet 8364045e3fa2441e8dc1a1d87f73361d
I20260810 08:00:29.631860 14689 catalog_manager.cc:6565] ReplaceTablet: tablet 8364045e3fa2441e8dc1a1d87f73361d deleted and replaced by tablet fd97cbfb16bf4ee2b4eed1e8bbbc7224
I20260810 08:00:29.642341 15640 consensus_queue.cc:1035] T e46648198f5343f7a65079887e5df159 P a91b27762182454baada71e91ceab224 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 08:00:29.578637 15478 ts_tablet_manager.cc:1918] T 9ddae63b29c34e1397c5765e52e1478f P 3fc5e568f3724f158b616c303efb9eae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 08:00:29.650339 15478 log.cc:1196] T 9ddae63b29c34e1397c5765e52e1478f P 3fc5e568f3724f158b616c303efb9eae: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal/wals/9ddae63b29c34e1397c5765e52e1478f
I20260810 08:00:29.651192 15478 ts_tablet_manager.cc:1939] T 9ddae63b29c34e1397c5765e52e1478f P 3fc5e568f3724f158b616c303efb9eae: Deleting consensus metadata
W20260810 08:00:29.653703 14681 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.14.57.194:36265, user_credentials={real_user=slave}} blocked reactor thread for 89001.9us
I20260810 08:00:29.654513 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet 35b6da60a5544d7e9b657a5f1e6f90ff (table default.Table_6e428ff2c4c7459b84d2882a089b490d [id=64f8570d723d4f5faba292689c535c01]) successfully deleted
I20260810 08:00:29.674908 15437 tablet_bootstrap.cc:492] T 37d8a37a16f34a0285c43c0d71b5f072 P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:29.675400 15437 ts_tablet_manager.cc:1397] T 37d8a37a16f34a0285c43c0d71b5f072 P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.457s user 0.008s sys 0.010s
I20260810 08:00:29.678028 15490 tablet_bootstrap.cc:492] T d22a778bfe784b2f8a7b4d2216909f10 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:29.678653 15490 ts_tablet_manager.cc:1397] T d22a778bfe784b2f8a7b4d2216909f10 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.312s user 0.010s sys 0.000s
I20260810 08:00:29.677829 15437 raft_consensus.cc:348] T 37d8a37a16f34a0285c43c0d71b5f072 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:29.702940 15234 leader_election.cc:302] T 49c35b7f4a72437fa0675abe8a8f3a71 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224; no voters:
I20260810 08:00:29.707991 15276 tablet_service.cc:1514] Processing DeleteTablet for tablet 8364045e3fa2441e8dc1a1d87f73361d with delete_type TABLET_DATA_DELETED (replaced by tablet fd97cbfb16bf4ee2b4eed1e8bbbc7224) from {username='slave'} at 127.0.0.1:40692
W20260810 08:00:29.716513 14691 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fd97cbfb16bf4ee2b4eed1e8bbbc7224 has no consensus state
W20260810 08:00:29.706804 14681 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.14.57.194:36265, user_credentials={real_user=slave}} blocked reactor thread for 49623.6us
I20260810 08:00:29.718815 15439 ts_tablet_manager.cc:1428] T 37d8a37a16f34a0285c43c0d71b5f072 P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.164s user 0.006s sys 0.000s
I20260810 08:00:29.720732 15439 tablet_bootstrap.cc:492] T fbf62f4eab8b49f282df3ab4d95d886f P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:29.721796 15447 raft_consensus.cc:2781] T 49c35b7f4a72437fa0675abe8a8f3a71 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Leader election won for term 1
W20260810 08:00:29.725749 15679 log.cc:924] Time spent T 5a22a48e2ba843b3a32a8900ab937ea0 P 3fc5e568f3724f158b616c303efb9eae: Append to log took a long time: real 0.072s user 0.001s sys 0.000s
I20260810 08:00:29.726752 15478 tablet_replica.cc:330] T 2faab63c5023431c98e7b01ce61e04c2 P 3fc5e568f3724f158b616c303efb9eae: stopping tablet replica
I20260810 08:00:29.729588 14678 catalog_manager.cc:4841] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): tablet 9ddae63b29c34e1397c5765e52e1478f (table default.Table_0e30c621046242609c9f6fc73b16d0e7 [id=e9608d66a9964bc6aafdf84f574fba02]) successfully deleted
I20260810 08:00:29.713342 15675 consensus_queue.cc:1035] T e46648198f5343f7a65079887e5df159 P a91b27762182454baada71e91ceab224 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 08:00:29.741982 15478 raft_consensus.cc:2229] T 2faab63c5023431c98e7b01ce61e04c2 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:29.742605 15478 raft_consensus.cc:2258] T 2faab63c5023431c98e7b01ce61e04c2 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:29.742785 15437 raft_consensus.cc:374] T 37d8a37a16f34a0285c43c0d71b5f072 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:29.743093 15437 raft_consensus.cc:729] T 37d8a37a16f34a0285c43c0d71b5f072 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:29.743705 15437 consensus_queue.cc:260] T 37d8a37a16f34a0285c43c0d71b5f072 P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:29.744683 15482 tablet_replica.cc:330] T 1461cdeda5664afd9c4c2b45199179b6 P a91b27762182454baada71e91ceab224: stopping tablet replica
I20260810 08:00:29.745406 15482 raft_consensus.cc:2229] T 1461cdeda5664afd9c4c2b45199179b6 P a91b27762182454baada71e91ceab224 [term 1 LEADER]: Raft consensus shutting down.
I20260810 08:00:29.746107 15478 ts_tablet_manager.cc:1905] T 2faab63c5023431c98e7b01ce61e04c2 P 3fc5e568f3724f158b616c303efb9eae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:29.746902 15482 raft_consensus.cc:2258] T 1461cdeda5664afd9c4c2b45199179b6 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 08:00:29.753811 14680 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet fd97cbfb16bf4ee2b4eed1e8bbbc7224 (table default.Table_290c3db6ca0e4fdf8e007f723b11d27d [id=97abcf4bf320496ebbec1fd2ed20765b], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet fd97cbfb16bf4ee2b4eed1e8bbbc7224 has no consensus state
I20260810 08:00:29.763916 15439 tablet_bootstrap.cc:654] T fbf62f4eab8b49f282df3ab4d95d886f P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:29.758749 15447 raft_consensus.cc:686] T 49c35b7f4a72437fa0675abe8a8f3a71 P 3fc5e568f3724f158b616c303efb9eae [term 1 LEADER]: Becoming Leader. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Running, Role: LEADER
I20260810 08:00:29.766038 15447 consensus_queue.cc:237] T 49c35b7f4a72437fa0675abe8a8f3a71 P 3fc5e568f3724f158b616c303efb9eae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:29.753268 15437 ts_tablet_manager.cc:1428] T 37d8a37a16f34a0285c43c0d71b5f072 P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.077s user 0.005s sys 0.000s
I20260810 08:00:29.772987 15437 tablet_bootstrap.cc:492] T fbf62f4eab8b49f282df3ab4d95d886f P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:29.774533 15694 tablet.cc:1722] T e46648198f5343f7a65079887e5df159 P 9d7219686f304f62a54acc7497cbbd07: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 08:00:29.789784 15482 ts_tablet_manager.cc:1905] T 1461cdeda5664afd9c4c2b45199179b6 P a91b27762182454baada71e91ceab224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:29.795010 15118 tablet_service.cc:1514] Processing DeleteTablet for tablet 8364045e3fa2441e8dc1a1d87f73361d with delete_type TABLET_DATA_DELETED (replaced by tablet fd97cbfb16bf4ee2b4eed1e8bbbc7224) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:29.795483 14962 tablet_service.cc:1514] Processing DeleteTablet for tablet 8364045e3fa2441e8dc1a1d87f73361d with delete_type TABLET_DATA_DELETED (replaced by tablet fd97cbfb16bf4ee2b4eed1e8bbbc7224) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:29.798554 15692 tablet.cc:1722] T e46648198f5343f7a65079887e5df159 P a91b27762182454baada71e91ceab224: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 08:00:29.681526 15490 raft_consensus.cc:348] T d22a778bfe784b2f8a7b4d2216909f10 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:29.799657 15490 raft_consensus.cc:374] T d22a778bfe784b2f8a7b4d2216909f10 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:29.799918 15490 raft_consensus.cc:729] T d22a778bfe784b2f8a7b4d2216909f10 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:29.800563 15490 consensus_queue.cc:260] T d22a778bfe784b2f8a7b4d2216909f10 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:29.799467 15437 tablet_bootstrap.cc:654] T fbf62f4eab8b49f282df3ab4d95d886f P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:29.824491 15437 tablet_bootstrap.cc:492] T fbf62f4eab8b49f282df3ab4d95d886f P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:29.825022 15437 ts_tablet_manager.cc:1397] T fbf62f4eab8b49f282df3ab4d95d886f P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.052s user 0.009s sys 0.004s
I20260810 08:00:29.831894 15439 tablet_bootstrap.cc:492] T fbf62f4eab8b49f282df3ab4d95d886f P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:29.832346 15439 ts_tablet_manager.cc:1397] T fbf62f4eab8b49f282df3ab4d95d886f P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.112s user 0.009s sys 0.002s
I20260810 08:00:29.835835 15437 raft_consensus.cc:348] T fbf62f4eab8b49f282df3ab4d95d886f P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:29.836489 15437 raft_consensus.cc:374] T fbf62f4eab8b49f282df3ab4d95d886f P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:29.836702 15437 raft_consensus.cc:729] T fbf62f4eab8b49f282df3ab4d95d886f P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:29.837297 15437 consensus_queue.cc:260] T fbf62f4eab8b49f282df3ab4d95d886f P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:29.838284 15490 ts_tablet_manager.cc:1428] T d22a778bfe784b2f8a7b4d2216909f10 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.159s user 0.006s sys 0.000s
I20260810 08:00:29.839214 15490 tablet_bootstrap.cc:492] T 68bc39707f44457fbf57c217ad827348 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:29.840461 15439 raft_consensus.cc:348] T fbf62f4eab8b49f282df3ab4d95d886f P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:29.841149 15439 raft_consensus.cc:374] T fbf62f4eab8b49f282df3ab4d95d886f P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:29.841441 15439 raft_consensus.cc:729] T fbf62f4eab8b49f282df3ab4d95d886f P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:29.843768 15437 ts_tablet_manager.cc:1428] T fbf62f4eab8b49f282df3ab4d95d886f P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20260810 08:00:29.844668 15437 tablet_bootstrap.cc:492] T d22a778bfe784b2f8a7b4d2216909f10 P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:29.845690 15439 consensus_queue.cc:260] T fbf62f4eab8b49f282df3ab4d95d886f P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:29.854691 15439 ts_tablet_manager.cc:1428] T fbf62f4eab8b49f282df3ab4d95d886f P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.022s user 0.007s sys 0.000s
I20260810 08:00:29.854651 15479 ts_tablet_manager.cc:1918] T 0c152aab7e244afd857534869b13815c P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 08:00:29.855032 15479 log.cc:1196] T 0c152aab7e244afd857534869b13815c P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/0c152aab7e244afd857534869b13815c
I20260810 08:00:29.855784 15479 ts_tablet_manager.cc:1939] T 0c152aab7e244afd857534869b13815c P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:29.860867 15439 tablet_bootstrap.cc:492] T d0761b090d8147b8a5537b0ed51108a9 P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
W20260810 08:00:29.874569 15234 consensus_peers.cc:487] T 35b6da60a5544d7e9b657a5f1e6f90ff P 3fc5e568f3724f158b616c303efb9eae -> Peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Couldn't send request to peer a91b27762182454baada71e91ceab224. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 35b6da60a5544d7e9b657a5f1e6f90ff. This is attempt 1: this message will repeat every 5th retry.
I20260810 08:00:29.814402 14681 catalog_manager.cc:4841] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): tablet a1c4f8f514c94128aedd07bd5efdc3be (table default.Table_89fe7ad723964551909b73ff6b72b9af [id=ce96e7fb7c46404a859ca08c1fd1b26a]) successfully deleted
W20260810 08:00:29.875540 14681 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.14.57.193:43601, user_credentials={real_user=slave}} blocked reactor thread for 61172.4us
W20260810 08:00:29.876158 14681 catalog_manager.cc:4821] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet 924c0f73638f48a88c08d45d459132e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 924c0f73638f48a88c08d45d459132e1 already in progress: creating tablet
I20260810 08:00:29.878041 14687 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_d1964b15134148e1a0a49f4ffe679821"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:29.880326 14687 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1964b15134148e1a0a49f4ffe679821 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 08:00:29.897387 15234 consensus_peers.cc:487] T 35b6da60a5544d7e9b657a5f1e6f90ff P 3fc5e568f3724f158b616c303efb9eae -> Peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Couldn't send request to peer 9d7219686f304f62a54acc7497cbbd07. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 35b6da60a5544d7e9b657a5f1e6f90ff. This is attempt 1: this message will repeat every 5th retry.
I20260810 08:00:29.898644 14691 master_service.cc:946] ReplaceTablet: received request to replace tablet d22a778bfe784b2f8a7b4d2216909f10 from {username='slave'} at 127.0.0.1:42300
I20260810 08:00:29.899346 14686 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_ae0fee60b7704372bf9ac04a22df6876" } modify_external_catalogs: true
I20260810 08:00:29.899981 14686 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_ae0fee60b7704372bf9ac04a22df6876" } modify_external_catalogs: true
I20260810 08:00:29.902474 14691 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_9a8eb842ebf244d5abbfb0f43373ca08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:29.904036 14691 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a8eb842ebf244d5abbfb0f43373ca08 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:29.927207 15699 tablet.cc:1722] T e46648198f5343f7a65079887e5df159 P 3fc5e568f3724f158b616c303efb9eae: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 08:00:29.931957 15490 tablet_bootstrap.cc:654] T 68bc39707f44457fbf57c217ad827348 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:29.933595 15617 raft_consensus.cc:482] T d22a778bfe784b2f8a7b4d2216909f10 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:29.937907 15617 raft_consensus.cc:504] T d22a778bfe784b2f8a7b4d2216909f10 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:29.940119 15617 leader_election.cc:288] T d22a778bfe784b2f8a7b4d2216909f10 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), a91b27762182454baada71e91ceab224 (127.14.57.193:43601)
I20260810 08:00:29.950316 15479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43002 (request call id 48) took 3180 ms. Trace:
I20260810 08:00:29.950652 15479 rpcz_store.cc:276] 0810 08:00:26.770302 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:26.770481 (+ 179us) service_pool.cc:227] Handling call
0810 08:00:29.376130 (+2605649us) ts_tablet_manager.cc:1145] Deleting tablet 0c152aab7e244afd857534869b13815c
0810 08:00:29.376180 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:29.633232 (+257052us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:29.854377 (+221145us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:29.950242 (+ 95865us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2605060}
I20260810 08:00:29.951756 15479 tablet_replica.cc:330] T 728f8f80b3bb4b10871d8b400f25377b P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:29.952395 15479 raft_consensus.cc:2229] T 728f8f80b3bb4b10871d8b400f25377b P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:29.952803 15479 raft_consensus.cc:2258] T 728f8f80b3bb4b10871d8b400f25377b P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:29.954355 15479 ts_tablet_manager.cc:1905] T 728f8f80b3bb4b10871d8b400f25377b P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:29.957304 14694 catalog_manager.cc:5495] T 5a22a48e2ba843b3a32a8900ab937ea0 P 9d7219686f304f62a54acc7497cbbd07 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194). New cstate: current_term: 1 leader_uuid: "9d7219686f304f62a54acc7497cbbd07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } health_report { overall_health: HEALTHY } } }
I20260810 08:00:29.959295 14694 catalog_manager.cc:5587] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265) does not have the latest schema for tablet 5a22a48e2ba843b3a32a8900ab937ea0 (table default.Table_5a6ab9b9acc9408c8811f77acaca9a0c [id=ac3a5787385940bca8030d7bac893a27]). Expected version 2 got 0
I20260810 08:00:29.960341 14694 catalog_manager.cc:5587] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265) does not have the latest schema for tablet 49c35b7f4a72437fa0675abe8a8f3a71 (table default.Table_1567aaa1d2ab44028745e82c29232682 [id=6cf63d3e931a4f6487074494254805bb]). Expected version 4 got 0
I20260810 08:00:29.963387 15302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d22a778bfe784b2f8a7b4d2216909f10" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
W20260810 08:00:29.972445 15076 leader_election.cc:341] T d22a778bfe784b2f8a7b4d2216909f10 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:29.973820 15478 ts_tablet_manager.cc:1918] T 2faab63c5023431c98e7b01ce61e04c2 P 3fc5e568f3724f158b616c303efb9eae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 08:00:29.974385 15478 log.cc:1196] T 2faab63c5023431c98e7b01ce61e04c2 P 3fc5e568f3724f158b616c303efb9eae: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal/wals/2faab63c5023431c98e7b01ce61e04c2
I20260810 08:00:29.975431 15478 ts_tablet_manager.cc:1939] T 2faab63c5023431c98e7b01ce61e04c2 P 3fc5e568f3724f158b616c303efb9eae: Deleting consensus metadata
I20260810 08:00:29.982129 14693 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet 3ac554bd37b045a2a8cf2743f4645f28
I20260810 08:00:29.983572 14686 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet e162a89dcf5a4890aa5a93dfeadd131e
I20260810 08:00:29.987311 15447 raft_consensus.cc:482] T fbf62f4eab8b49f282df3ab4d95d886f P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:29.987836 15447 raft_consensus.cc:504] T fbf62f4eab8b49f282df3ab4d95d886f P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:29.990072 15447 leader_election.cc:288] T fbf62f4eab8b49f282df3ab4d95d886f P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:29.992355 15276 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ac554bd37b045a2a8cf2743f4645f28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:29 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:29.992537 15147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf62f4eab8b49f282df3ab4d95d886f" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:29.993183 15272 tablet_service.cc:1514] Processing DeleteTablet for tablet e162a89dcf5a4890aa5a93dfeadd131e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:29 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:29.993180 15147 raft_consensus.cc:2454] T fbf62f4eab8b49f282df3ab4d95d886f P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 0.
I20260810 08:00:29.996466 15234 leader_election.cc:302] T fbf62f4eab8b49f282df3ab4d95d886f P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, 9d7219686f304f62a54acc7497cbbd07; no voters:
I20260810 08:00:29.997179 15447 raft_consensus.cc:2781] T fbf62f4eab8b49f282df3ab4d95d886f P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:29.997512 15447 raft_consensus.cc:482] T fbf62f4eab8b49f282df3ab4d95d886f P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:29.997890 15447 raft_consensus.cc:3037] T fbf62f4eab8b49f282df3ab4d95d886f P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:30.003556 15272 tablet_service.cc:1467] Processing CreateTablet for tablet 3855a2b4d90a47ad918c7a90cc8c7e4a (DEFAULT_TABLE table=default.Table_e6beb01e821446feaced3468aaa20563 [id=dd2a9195f7ae4ee98b6c71985d2658bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.004901 15272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3855a2b4d90a47ad918c7a90cc8c7e4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.006343 15276 tablet_service.cc:1467] Processing CreateTablet for tablet 0687d323788642f798465ece973702bb (DEFAULT_TABLE table=default.Table_44d94cfdac894ff68cfbb8984912231c [id=0d60b763c4804818ae26fa693a4fd808]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.007299 15276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0687d323788642f798465ece973702bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.019480 15617 raft_consensus.cc:482] T 2d295305b5a843eaa84c8ad666355b84 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:30.019999 15617 raft_consensus.cc:504] T 2d295305b5a843eaa84c8ad666355b84 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:30.022984 15617 leader_election.cc:288] T 2d295305b5a843eaa84c8ad666355b84 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), a91b27762182454baada71e91ceab224 (127.14.57.193:43601)
I20260810 08:00:30.022464 15439 tablet_bootstrap.cc:654] T d0761b090d8147b8a5537b0ed51108a9 P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:30.034121 14690 catalog_manager.cc:5587] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729) does not have the latest schema for tablet 5a22a48e2ba843b3a32a8900ab937ea0 (table default.Table_5a6ab9b9acc9408c8811f77acaca9a0c [id=ac3a5787385940bca8030d7bac893a27]). Expected version 2 got 0
I20260810 08:00:30.035282 14690 catalog_manager.cc:5587] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729) does not have the latest schema for tablet 49c35b7f4a72437fa0675abe8a8f3a71 (table default.Table_1567aaa1d2ab44028745e82c29232682 [id=6cf63d3e931a4f6487074494254805bb]). Expected version 4 got 0
I20260810 08:00:30.036214 14690 catalog_manager.cc:5587] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729) does not have the latest schema for tablet e46648198f5343f7a65079887e5df159 (table default.Table_d81fe7cc77ad45d39c8b44e1299dbefc [id=a705653aa3484a69b369e7e39273d28e]). Expected version 1 got 0
I20260810 08:00:30.043927 14689 catalog_manager.cc:5587] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601) does not have the latest schema for tablet 49c35b7f4a72437fa0675abe8a8f3a71 (table default.Table_1567aaa1d2ab44028745e82c29232682 [id=6cf63d3e931a4f6487074494254805bb]). Expected version 4 got 0
I20260810 08:00:30.045287 14689 catalog_manager.cc:5587] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601) does not have the latest schema for tablet 5a22a48e2ba843b3a32a8900ab937ea0 (table default.Table_5a6ab9b9acc9408c8811f77acaca9a0c [id=ac3a5787385940bca8030d7bac893a27]). Expected version 2 got 0
I20260810 08:00:30.047641 15439 tablet_bootstrap.cc:492] T d0761b090d8147b8a5537b0ed51108a9 P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:30.048100 15439 ts_tablet_manager.cc:1397] T d0761b090d8147b8a5537b0ed51108a9 P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.187s user 0.008s sys 0.004s
I20260810 08:00:30.050807 15439 raft_consensus.cc:348] T d0761b090d8147b8a5537b0ed51108a9 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:30.051553 15439 raft_consensus.cc:374] T d0761b090d8147b8a5537b0ed51108a9 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:30.051795 15439 raft_consensus.cc:729] T d0761b090d8147b8a5537b0ed51108a9 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:30.052389 15439 consensus_queue.cc:260] T d0761b090d8147b8a5537b0ed51108a9 P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:30.058043 15478 tablet_replica.cc:330] T 35b6da60a5544d7e9b657a5f1e6f90ff P 3fc5e568f3724f158b616c303efb9eae: stopping tablet replica
I20260810 08:00:30.058197 15439 ts_tablet_manager.cc:1428] T d0761b090d8147b8a5537b0ed51108a9 P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
I20260810 08:00:30.058802 15478 raft_consensus.cc:2229] T 35b6da60a5544d7e9b657a5f1e6f90ff P 3fc5e568f3724f158b616c303efb9eae [term 1 LEADER]: Raft consensus shutting down.
I20260810 08:00:30.059137 15439 tablet_bootstrap.cc:492] T 2d295305b5a843eaa84c8ad666355b84 P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:30.059587 15478 pending_rounds.cc:62] T 35b6da60a5544d7e9b657a5f1e6f90ff P 3fc5e568f3724f158b616c303efb9eae: Trying to abort 1 pending ops.
I20260810 08:00:30.059839 15478 pending_rounds.cc:68] T 35b6da60a5544d7e9b657a5f1e6f90ff P 3fc5e568f3724f158b616c303efb9eae: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316884803019530240 op_type: NO_OP noop_request { }
I20260810 08:00:30.060245 15478 raft_consensus.cc:2866] T 35b6da60a5544d7e9b657a5f1e6f90ff P 3fc5e568f3724f158b616c303efb9eae [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260810 08:00:30.060559 15478 raft_consensus.cc:2258] T 35b6da60a5544d7e9b657a5f1e6f90ff P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 08:00:30.061734 14681 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.57.252:33819, user_credentials={real_user=slave}} blocked reactor thread for 68013.9us
W20260810 08:00:30.061403 14688 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.14.57.193:35941 (request call id 32) took 2178 ms (client timeout 1000 ms). Trace:
I20260810 08:00:30.064460 15118 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ac554bd37b045a2a8cf2743f4645f28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:29 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:30.065153 15122 tablet_service.cc:1514] Processing DeleteTablet for tablet e162a89dcf5a4890aa5a93dfeadd131e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:29 UTC) from {username='slave'} at 127.0.0.1:43002
W20260810 08:00:30.066623 14688 rpcz_store.cc:269] 0810 08:00:27.883329 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:27.883523 (+ 194us) service_pool.cc:227] Handling call
0810 08:00:30.061215 (+2177692us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":6}
I20260810 08:00:30.069083 15277 tablet_service.cc:1467] Processing CreateTablet for tablet 5f2e51d10e7142938e03b2ef687b0edd (DEFAULT_TABLE table=default.Table_4f0bc756581c45958dc46729a9430f79 [id=87b8720fcdf143299d050c9546dcd91d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.070294 15277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f2e51d10e7142938e03b2ef687b0edd. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 08:00:30.071178 14695 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.14.57.193:35941 (request call id 33) took 1181 ms (client timeout 1000 ms). Trace:
I20260810 08:00:30.071446 15447 raft_consensus.cc:504] T fbf62f4eab8b49f282df3ab4d95d886f P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:30.073326 15447 leader_election.cc:288] T fbf62f4eab8b49f282df3ab4d95d886f P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Requested vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:30.073757 15479 ts_tablet_manager.cc:1918] T 728f8f80b3bb4b10871d8b400f25377b P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20260810 08:00:30.071487 14695 rpcz_store.cc:269] 0810 08:00:28.889815 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:29.023552 (+133737us) service_pool.cc:227] Handling call
0810 08:00:29.041942 (+ 18390us) spinlock_profiling.cc:243] Waited 16.5 ms on lock 0x7b5400021554. stack: 00007fe2372c4033 00007fe2372c3f30 00007fe236e45534 00000000004db2b4 00000000004db249 00007fe23e1d0af0 00007fe23e253f2a 00007fe24011fd6c 00007fe240219c3e 00007fe23dd50fb4 00007fe23dd50f42 00007fe23dd50e71 00007fe23dd50dec 00007fe23dd500a2 00007fe23bffa94c 00007fe23bff9d86
0810 08:00:30.071002 (+1029060us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 08:00:29.463267 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 08:00:29.463677 (+ 410us) write_op.cc:432] Acquiring schema lock in shared mode
0810 08:00:29.463704 (+ 27us) write_op.cc:435] Acquired schema lock
0810 08:00:29.463719 (+ 15us) tablet.cc:660] Decoding operations
0810 08:00:29.463993 (+ 274us) write_op.cc:620] Acquiring the partition lock for write op
0810 08:00:29.464038 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0810 08:00:29.464063 (+ 25us) tablet.cc:683] Acquiring locks for 1 operations
0810 08:00:29.464214 (+ 151us) tablet.cc:699] Row locks acquired
0810 08:00:29.464229 (+ 15us) write_op.cc:260] PREPARE: finished
0810 08:00:29.464355 (+ 126us) write_op.cc:270] Start()
0810 08:00:29.464449 (+ 94us) write_op.cc:276] Timestamp: P: 1786348829464272 usec, L: 0
0810 08:00:29.464465 (+ 16us) op_driver.cc:331] REPLICATION: starting
0810 08:00:29.465017 (+ 552us) log.cc:842] Serialized 533 byte log entry
0810 08:00:29.568586 (+103569us) op_driver.cc:444] REPLICATION: finished
0810 08:00:29.569082 (+ 496us) write_op.cc:301] APPLY: starting
0810 08:00:29.569157 (+ 75us) tablet.cc:1365] starting BulkCheckPresence
0810 08:00:29.569281 (+ 124us) tablet.cc:1368] finished BulkCheckPresence
0810 08:00:29.569410 (+ 129us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 08:00:29.569723 (+ 313us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 08:00:29.569970 (+ 247us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 08:00:29.570036 (+ 66us) write_op.cc:312] APPLY: finished
0810 08:00:29.570379 (+ 343us) log.cc:842] Serialized 23 byte log entry
0810 08:00:29.570536 (+ 157us) write_op.cc:489] Releasing partition, row and schema locks
0810 08:00:29.570637 (+ 101us) write_op.cc:454] Released schema lock
0810 08:00:29.570671 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0810 08:00:29.570924 (+ 253us) write_op.cc:489] Releasing partition, row and schema locks
0810 08:00:29.570954 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":24686,"reported_tablets":9,"spinlock_wait_cycles":37963648,"child_traces":[["op",{"apply.queue_time_us":283,"apply.run_cpu_time_us":2068,"apply.run_wall_time_us":2068,"num_ops":1,"prepare.queue_time_us":360312,"prepare.run_cpu_time_us":2352,"prepare.run_wall_time_us":116904,"raft.queue_time_us":1558,"raft.run_cpu_time_us":319,"raft.run_wall_time_us":319,"replication_time_us":103965}]]}
I20260810 08:00:30.074457 15479 log.cc:1196] T 728f8f80b3bb4b10871d8b400f25377b P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/728f8f80b3bb4b10871d8b400f25377b
I20260810 08:00:30.074549 15278 tablet_service.cc:1467] Processing CreateTablet for tablet 2fbda1b3b5b4482ba6f03f3e275d94e7 (DEFAULT_TABLE table=default.Table_fea0f565d8a0450a91e8729141f89d99 [id=3ffa6d723f2344eaac1c5f831d3320a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.068632 14678 catalog_manager.cc:4841] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): tablet 2faab63c5023431c98e7b01ce61e04c2 (table default.Table_10462271d83b4063b56a4cd0c8b6f94a [id=67e4dae7d49b48669b176960841b25ab]) successfully deleted
I20260810 08:00:30.075687 15278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fbda1b3b5b4482ba6f03f3e275d94e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.075760 15479 ts_tablet_manager.cc:1939] T 728f8f80b3bb4b10871d8b400f25377b P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:30.077209 15279 tablet_service.cc:1467] Processing CreateTablet for tablet 8a0e1ad5653f46b4861b3e39629fbd2c (DEFAULT_TABLE table=default.Table_3d528c5b4aa54991bf128ada67924546 [id=8c493c9dfd3f4f358bc02f33869f3a16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.078207 15279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a0e1ad5653f46b4861b3e39629fbd2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.064060 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet 0c152aab7e244afd857534869b13815c (table default.Table_8ac532f4969e45a197db311de2a7b795 [id=7893824f2c374508b7b3c4635ebea5b6]) successfully deleted
I20260810 08:00:30.080339 15280 tablet_service.cc:1467] Processing CreateTablet for tablet 4ca688b4031c41a9884829fbd0be1f33 (DEFAULT_TABLE table=default.Table_357e1765e09e4c4ab104ee0436bc1091 [id=da2bb7febfe9421380f61df1393ea3b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.080972 15122 tablet_service.cc:1467] Processing CreateTablet for tablet 3855a2b4d90a47ad918c7a90cc8c7e4a (DEFAULT_TABLE table=default.Table_e6beb01e821446feaced3468aaa20563 [id=dd2a9195f7ae4ee98b6c71985d2658bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.081413 15280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ca688b4031c41a9884829fbd0be1f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.082082 15122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3855a2b4d90a47ad918c7a90cc8c7e4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.082924 14692 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:42300 (request call id 134) took 1465 ms. Trace:
I20260810 08:00:30.083612 15274 tablet_service.cc:1467] Processing CreateTablet for tablet 0a60001d76604d5085c843961d52213f (DEFAULT_TABLE table=default.Table_ebe6adac96b74785b939b2a0d8e7a192 [id=f2c914866c3f418caf5e08a2ad03a91b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.084621 15274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a60001d76604d5085c843961d52213f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.083204 14692 rpcz_store.cc:276] 0810 08:00:28.617256 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:28.718928 (+101672us) service_pool.cc:227] Handling call
0810 08:00:28.750981 (+ 32053us) spinlock_profiling.cc:243] Waited 31.3 ms on lock 0x7b5400021554. stack: 00007fe2372c4033 00007fe2372c3f30 00007fe236e45534 00000000004db2b4 00000000004db249 00007fe23e1d0af0 00007fe23e253f2a 00007fe24011fd6c 00007fe24021b7d6 00007fe23dd61634 00007fe23dd615c2 00007fe23dd614f1 00007fe23dd6146c 00007fe23dd60722 00007fe23bffa94c 00007fe23bff9d86
0810 08:00:28.751815 (+ 834us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 08:00:28.752450 (+ 635us) catalog_manager.cc:3616] Apply alter schema
0810 08:00:28.753000 (+ 550us) catalog_manager.cc:3639] Acquired catalog manager lock
0810 08:00:28.753252 (+ 252us) catalog_manager.cc:3786] Updating metadata on disk
0810 08:00:29.292927 (+539675us) catalog_manager.cc:3817] Committing alterations to in-memory state
0810 08:00:30.082735 (+789808us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 08:00:29.242914 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 08:00:29.243485 (+ 571us) write_op.cc:432] Acquiring schema lock in shared mode
0810 08:00:29.243533 (+ 48us) write_op.cc:435] Acquired schema lock
0810 08:00:29.243545 (+ 12us) tablet.cc:660] Decoding operations
0810 08:00:29.243979 (+ 434us) write_op.cc:620] Acquiring the partition lock for write op
0810 08:00:29.244081 (+ 102us) write_op.cc:641] Partition lock acquired for write op
0810 08:00:29.244103 (+ 22us) tablet.cc:683] Acquiring locks for 1 operations
0810 08:00:29.280297 (+ 36194us) tablet.cc:699] Row locks acquired
0810 08:00:29.280313 (+ 16us) write_op.cc:260] PREPARE: finished
0810 08:00:29.280437 (+ 124us) write_op.cc:270] Start()
0810 08:00:29.280523 (+ 86us) write_op.cc:276] Timestamp: P: 1786348829280376 usec, L: 0
0810 08:00:29.280538 (+ 15us) op_driver.cc:331] REPLICATION: starting
0810 08:00:29.281155 (+ 617us) log.cc:842] Serialized 405 byte log entry
0810 08:00:29.290587 (+ 9432us) op_driver.cc:444] REPLICATION: finished
0810 08:00:29.291019 (+ 432us) write_op.cc:301] APPLY: starting
0810 08:00:29.291095 (+ 76us) tablet.cc:1365] starting BulkCheckPresence
0810 08:00:29.291224 (+ 129us) tablet.cc:1368] finished BulkCheckPresence
0810 08:00:29.291350 (+ 126us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 08:00:29.291554 (+ 204us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 08:00:29.291732 (+ 178us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 08:00:29.291794 (+ 62us) write_op.cc:312] APPLY: finished
0810 08:00:29.292056 (+ 262us) log.cc:842] Serialized 23 byte log entry
0810 08:00:29.292241 (+ 185us) write_op.cc:489] Releasing partition, row and schema locks
0810 08:00:29.292320 (+ 79us) write_op.cc:454] Released schema lock
0810 08:00:29.292356 (+ 36us) write_op.cc:341] FINISH: Updating metrics
0810 08:00:29.292544 (+ 188us) write_op.cc:489] Releasing partition, row and schema locks
0810 08:00:29.292577 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":7156,"spinlock_wait_cycles":72088960,"child_traces":[["op",{"apply.queue_time_us":229,"apply.run_cpu_time_us":1743,"apply.run_wall_time_us":1744,"num_ops":1,"prepare.queue_time_us":488408,"prepare.run_cpu_time_us":2294,"prepare.run_wall_time_us":65463,"raft.queue_time_us":100,"raft.run_cpu_time_us":737,"raft.run_wall_time_us":737,"replication_time_us":9975}]]}
I20260810 08:00:30.086520 15273 tablet_service.cc:1467] Processing CreateTablet for tablet e162a89dcf5a4890aa5a93dfeadd131e (DEFAULT_TABLE table=default.Table_ae0fee60b7704372bf9ac04a22df6876 [id=bdcacd958d574265adebc1fd1cb6c1fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.087291 15640 raft_consensus.cc:482] T fbf62f4eab8b49f282df3ab4d95d886f P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:30.087613 15273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e162a89dcf5a4890aa5a93dfeadd131e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.087582 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d22a778bfe784b2f8a7b4d2216909f10" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:30.087725 15640 raft_consensus.cc:504] T fbf62f4eab8b49f282df3ab4d95d886f P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:30.089016 15275 tablet_service.cc:1467] Processing CreateTablet for tablet 68048e2768cf441d906e675254175c9c (DEFAULT_TABLE table=default.Table_4ff94124f5cb4710a0ad3ff1781efcd3 [id=0b1956596de94b569ebd9f1816e4b699]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.090070 15275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68048e2768cf441d906e675254175c9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.092377 15271 tablet_service.cc:1467] Processing CreateTablet for tablet a4a78b2a960d4d78aa1f48a9c9edf3be (DEFAULT_TABLE table=default.Table_89fe7ad723964551909b73ff6b72b9af [id=ce96e7fb7c46404a859ca08c1fd1b26a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.093552 15271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4a78b2a960d4d78aa1f48a9c9edf3be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.096256 15121 tablet_service.cc:1467] Processing CreateTablet for tablet 5f2e51d10e7142938e03b2ef687b0edd (DEFAULT_TABLE table=default.Table_4f0bc756581c45958dc46729a9430f79 [id=87b8720fcdf143299d050c9546dcd91d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.098917 15119 tablet_service.cc:1467] Processing CreateTablet for tablet 2fbda1b3b5b4482ba6f03f3e275d94e7 (DEFAULT_TABLE table=default.Table_fea0f565d8a0450a91e8729141f89d99 [id=3ffa6d723f2344eaac1c5f831d3320a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.100986 15118 tablet_service.cc:1467] Processing CreateTablet for tablet 0687d323788642f798465ece973702bb (DEFAULT_TABLE table=default.Table_44d94cfdac894ff68cfbb8984912231c [id=0d60b763c4804818ae26fa693a4fd808]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.105162 15121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f2e51d10e7142938e03b2ef687b0edd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.110352 15119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fbda1b3b5b4482ba6f03f3e275d94e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.118887 15118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0687d323788642f798465ece973702bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.131855 15270 tablet_service.cc:1467] Processing CreateTablet for tablet 3281dd757ff44d9498ef70ad372eb150 (DEFAULT_TABLE table=default.Table_409addd050ee45b981bddfacfab24c03 [id=76166bd7cb584067b7d2e305603a840b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.133471 15270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3281dd757ff44d9498ef70ad372eb150. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.139739 15269 tablet_service.cc:1467] Processing CreateTablet for tablet 655fd33901974b9c85f52f92ff677247 (DEFAULT_TABLE table=default.Table_69ee062d338749dbba588a79aee54e1e [id=c80b653f9aec49baa511433216b67e29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.141444 15269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 655fd33901974b9c85f52f92ff677247. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.146437 15125 tablet_service.cc:1467] Processing CreateTablet for tablet 8a0e1ad5653f46b4861b3e39629fbd2c (DEFAULT_TABLE table=default.Table_3d528c5b4aa54991bf128ada67924546 [id=8c493c9dfd3f4f358bc02f33869f3a16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.147702 15125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a0e1ad5653f46b4861b3e39629fbd2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.155440 15123 tablet_service.cc:1467] Processing CreateTablet for tablet 4ca688b4031c41a9884829fbd0be1f33 (DEFAULT_TABLE table=default.Table_357e1765e09e4c4ab104ee0436bc1091 [id=da2bb7febfe9421380f61df1393ea3b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.156563 15120 tablet_service.cc:1467] Processing CreateTablet for tablet e162a89dcf5a4890aa5a93dfeadd131e (DEFAULT_TABLE table=default.Table_ae0fee60b7704372bf9ac04a22df6876 [id=bdcacd958d574265adebc1fd1cb6c1fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.156883 15123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ca688b4031c41a9884829fbd0be1f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.156845 15116 tablet_service.cc:1467] Processing CreateTablet for tablet a4a78b2a960d4d78aa1f48a9c9edf3be (DEFAULT_TABLE table=default.Table_89fe7ad723964551909b73ff6b72b9af [id=ce96e7fb7c46404a859ca08c1fd1b26a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.158389 15117 tablet_service.cc:1467] Processing CreateTablet for tablet 68048e2768cf441d906e675254175c9c (DEFAULT_TABLE table=default.Table_4ff94124f5cb4710a0ad3ff1781efcd3 [id=0b1956596de94b569ebd9f1816e4b699]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.158401 15126 tablet_service.cc:1467] Processing CreateTablet for tablet 0a60001d76604d5085c843961d52213f (DEFAULT_TABLE table=default.Table_ebe6adac96b74785b939b2a0d8e7a192 [id=f2c914866c3f418caf5e08a2ad03a91b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.159684 15114 tablet_service.cc:1467] Processing CreateTablet for tablet 3281dd757ff44d9498ef70ad372eb150 (DEFAULT_TABLE table=default.Table_409addd050ee45b981bddfacfab24c03 [id=76166bd7cb584067b7d2e305603a840b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.160389 15120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e162a89dcf5a4890aa5a93dfeadd131e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.161013 15116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4a78b2a960d4d78aa1f48a9c9edf3be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.161738 15126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a60001d76604d5085c843961d52213f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.164476 15114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3281dd757ff44d9498ef70ad372eb150. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.165923 15117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68048e2768cf441d906e675254175c9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.194841 15268 tablet_service.cc:1467] Processing CreateTablet for tablet b18980bffe6045b5b69bcecb47b4c711 (DEFAULT_TABLE table=default.Table_64e1c230e9624fc9ba6c328346150ffe [id=46e6c2115d8c4b5e9907ed76b273638a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.196449 15268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b18980bffe6045b5b69bcecb47b4c711. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.200700 15113 tablet_service.cc:1467] Processing CreateTablet for tablet 655fd33901974b9c85f52f92ff677247 (DEFAULT_TABLE table=default.Table_69ee062d338749dbba588a79aee54e1e [id=c80b653f9aec49baa511433216b67e29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.202011 15113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 655fd33901974b9c85f52f92ff677247. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.211306 15479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43002 (request call id 59) took 2391 ms. Trace:
I20260810 08:00:30.229046 15478 ts_tablet_manager.cc:1905] T 35b6da60a5544d7e9b657a5f1e6f90ff P 3fc5e568f3724f158b616c303efb9eae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:30.231086 15267 tablet_service.cc:1467] Processing CreateTablet for tablet e1b42eac370145d4b684456639968485 (DEFAULT_TABLE table=default.Table_7f3a4e19ded64b3783a00b3d4c6839c3 [id=6c6afcf578534997878b98feb9574e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.232555 15439 tablet_bootstrap.cc:654] T 2d295305b5a843eaa84c8ad666355b84 P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
W20260810 08:00:30.232972 15234 consensus_peers.cc:487] T 114d3eac994f466bb1787e066de05e2c P 3fc5e568f3724f158b616c303efb9eae -> Peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Couldn't send request to peer 9d7219686f304f62a54acc7497cbbd07. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 114d3eac994f466bb1787e066de05e2c. This is attempt 1: this message will repeat every 5th retry.
I20260810 08:00:30.232810 15302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d295305b5a843eaa84c8ad666355b84" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:30.238391 15267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1b42eac370145d4b684456639968485. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.211374 15112 tablet_service.cc:1467] Processing CreateTablet for tablet b18980bffe6045b5b69bcecb47b4c711 (DEFAULT_TABLE table=default.Table_64e1c230e9624fc9ba6c328346150ffe [id=46e6c2115d8c4b5e9907ed76b273638a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.232144 15147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf62f4eab8b49f282df3ab4d95d886f" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07"
I20260810 08:00:30.259194 15479 rpcz_store.cc:276] 0810 08:00:27.820032 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:27.820197 (+ 165us) service_pool.cc:227] Handling call
0810 08:00:29.951610 (+2131413us) ts_tablet_manager.cc:1145] Deleting tablet 728f8f80b3bb4b10871d8b400f25377b
0810 08:00:29.951648 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:30.049914 (+ 98266us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:30.073331 (+ 23417us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:30.211225 (+137894us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2130560}
I20260810 08:00:30.265166 15439 tablet_bootstrap.cc:492] T 2d295305b5a843eaa84c8ad666355b84 P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:30.265743 15439 ts_tablet_manager.cc:1397] T 2d295305b5a843eaa84c8ad666355b84 P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.207s user 0.002s sys 0.010s
I20260810 08:00:30.275154 15479 tablet_replica.cc:330] T 2cca8a18778c49d0be01363c31ef180b P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:30.268667 15439 raft_consensus.cc:348] T 2d295305b5a843eaa84c8ad666355b84 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:30.278330 15439 raft_consensus.cc:374] T 2d295305b5a843eaa84c8ad666355b84 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:30.279269 15479 raft_consensus.cc:2229] T 2cca8a18778c49d0be01363c31ef180b P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:30.280635 15439 raft_consensus.cc:729] T 2d295305b5a843eaa84c8ad666355b84 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:30.282830 15479 raft_consensus.cc:2258] T 2cca8a18778c49d0be01363c31ef180b P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:30.289170 15640 leader_election.cc:288] T fbf62f4eab8b49f282df3ab4d95d886f P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:30.283565 15439 consensus_queue.cc:260] T 2d295305b5a843eaa84c8ad666355b84 P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:30.292126 15439 ts_tablet_manager.cc:1428] T 2d295305b5a843eaa84c8ad666355b84 P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.026s user 0.003s sys 0.002s
I20260810 08:00:30.293186 15439 tablet_bootstrap.cc:492] T 25de7bf64bf6450ba17b7748568bc0ed P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
W20260810 08:00:30.295516 14918 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.57.195:37729, user_credentials={real_user=slave}} blocked reactor thread for 206893us
I20260810 08:00:30.306819 15300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf62f4eab8b49f282df3ab4d95d886f" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:30.307736 15300 raft_consensus.cc:2379] T fbf62f4eab8b49f282df3ab4d95d886f P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a91b27762182454baada71e91ceab224 in current term 1: Already voted for candidate 3fc5e568f3724f158b616c303efb9eae in this term.
I20260810 08:00:30.296484 15437 tablet_bootstrap.cc:700] T d22a778bfe784b2f8a7b4d2216909f10 P a91b27762182454baada71e91ceab224: Time spent opening tablet: real 0.210s user 0.010s sys 0.003s
I20260810 08:00:30.296571 14962 tablet_service.cc:1467] Processing CreateTablet for tablet 3855a2b4d90a47ad918c7a90cc8c7e4a (DEFAULT_TABLE table=default.Table_e6beb01e821446feaced3468aaa20563 [id=dd2a9195f7ae4ee98b6c71985d2658bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.297555 14966 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ac554bd37b045a2a8cf2743f4645f28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:29 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:30.309448 14962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3855a2b4d90a47ad918c7a90cc8c7e4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.323309 14988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf62f4eab8b49f282df3ab4d95d886f" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:30.323988 14988 raft_consensus.cc:2454] T fbf62f4eab8b49f282df3ab4d95d886f P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 0.
W20260810 08:00:30.325182 14694 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.14.57.194:55587 (request call id 42) took 1137 ms (client timeout 1000 ms). Trace:
I20260810 08:00:30.327140 14988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf62f4eab8b49f282df3ab4d95d886f" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224"
I20260810 08:00:30.327641 14988 raft_consensus.cc:3037] T fbf62f4eab8b49f282df3ab4d95d886f P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:30.327687 15479 ts_tablet_manager.cc:1905] T 2cca8a18778c49d0be01363c31ef180b P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:30.331612 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet 728f8f80b3bb4b10871d8b400f25377b (table default.Table_4c0b6e5e875f43e088f4f0215cf39284 [id=b89eea0100b945c1851d323ce8c97591]) successfully deleted
I20260810 08:00:30.334540 14988 raft_consensus.cc:2454] T fbf62f4eab8b49f282df3ab4d95d886f P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 1.
I20260810 08:00:30.338359 15439 tablet_bootstrap.cc:654] T 25de7bf64bf6450ba17b7748568bc0ed P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:30.356771 15439 tablet_bootstrap.cc:492] T 25de7bf64bf6450ba17b7748568bc0ed P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:30.357417 15439 ts_tablet_manager.cc:1397] T 25de7bf64bf6450ba17b7748568bc0ed P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.064s user 0.010s sys 0.000s
W20260810 08:00:30.325523 14694 rpcz_store.cc:269] 0810 08:00:29.187340 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:29.187480 (+ 140us) service_pool.cc:227] Handling call
0810 08:00:29.250143 (+ 62663us) spinlock_profiling.cc:243] Waited 60.5 ms on lock 0x7b5400021554. stack: 00007fe2372c4033 00007fe2372c3f30 00007fe236e45534 00000000004db2b4 00000000004db249 00007fe23e1d0af0 00007fe23e253f2a 00007fe24011fd6c 00007fe240219c3e 00007fe23dd50fb4 00007fe23dd50f42 00007fe23dd50e71 00007fe23dd50dec 00007fe23dd500a2 00007fe23bffa94c 00007fe23bff9d86
0810 08:00:30.325110 (+1074967us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 08:00:29.990342 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 08:00:29.990710 (+ 368us) write_op.cc:432] Acquiring schema lock in shared mode
0810 08:00:29.990731 (+ 21us) write_op.cc:435] Acquired schema lock
0810 08:00:29.990752 (+ 21us) tablet.cc:660] Decoding operations
0810 08:00:29.991052 (+ 300us) write_op.cc:620] Acquiring the partition lock for write op
0810 08:00:29.991090 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0810 08:00:29.991110 (+ 20us) tablet.cc:683] Acquiring locks for 1 operations
0810 08:00:29.991252 (+ 142us) tablet.cc:699] Row locks acquired
0810 08:00:29.991264 (+ 12us) write_op.cc:260] PREPARE: finished
0810 08:00:29.991360 (+ 96us) write_op.cc:270] Start()
0810 08:00:29.991451 (+ 91us) write_op.cc:276] Timestamp: P: 1786348829991226 usec, L: 0
0810 08:00:29.991470 (+ 19us) op_driver.cc:331] REPLICATION: starting
0810 08:00:30.002922 (+ 11452us) log.cc:842] Serialized 533 byte log entry
0810 08:00:30.004756 (+ 1834us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b5400021554. stack: 00007fe2372c4033 00007fe2372c3f30 00007fe236e45534 00000000004db2b4 00000000004db249 00007fe23e1d0471 00007fe23e247198 00007fe23e6a8ade 00007fe23e6a82b8 00007fe23e6acba1 00007fe23e6acb59 00007fe23e6acae9 00007fe23e6acab1 00007fe23e6abdad 0000000000580784 00000000005805b9
0810 08:00:30.014828 (+ 10072us) op_driver.cc:444] REPLICATION: finished
0810 08:00:30.019862 (+ 5034us) write_op.cc:301] APPLY: starting
0810 08:00:30.019916 (+ 54us) tablet.cc:1365] starting BulkCheckPresence
0810 08:00:30.020006 (+ 90us) tablet.cc:1368] finished BulkCheckPresence
0810 08:00:30.020029 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 08:00:30.020210 (+ 181us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 08:00:30.020347 (+ 137us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 08:00:30.020389 (+ 42us) write_op.cc:312] APPLY: finished
0810 08:00:30.020561 (+ 172us) log.cc:842] Serialized 23 byte log entry
0810 08:00:30.020710 (+ 149us) write_op.cc:489] Releasing partition, row and schema locks
0810 08:00:30.020767 (+ 57us) write_op.cc:454] Released schema lock
0810 08:00:30.020805 (+ 38us) write_op.cc:341] FINISH: Updating metrics
0810 08:00:30.020969 (+ 164us) write_op.cc:489] Releasing partition, row and schema locks
0810 08:00:30.020991 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":9,"spinlock_wait_cycles":139240704,"child_traces":[["op",{"apply.queue_time_us":4843,"apply.run_cpu_time_us":1297,"apply.run_wall_time_us":1298,"num_ops":1,"prepare.queue_time_us":27916,"prepare.run_cpu_time_us":2637,"prepare.run_wall_time_us":14605,"raft.queue_time_us":172,"raft.run_cpu_time_us":700,"raft.run_wall_time_us":16711,"replication_time_us":23269,"spinlock_wait_cycles":24928512}]]}
I20260810 08:00:30.359717 15439 raft_consensus.cc:348] T 25de7bf64bf6450ba17b7748568bc0ed P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:30.360353 15439 raft_consensus.cc:374] T 25de7bf64bf6450ba17b7748568bc0ed P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:30.360587 15439 raft_consensus.cc:729] T 25de7bf64bf6450ba17b7748568bc0ed P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:30.323350 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d295305b5a843eaa84c8ad666355b84" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:30.370543 15437 tablet_bootstrap.cc:654] T d22a778bfe784b2f8a7b4d2216909f10 P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
W20260810 08:00:30.359634 14681 consensus_peers.cc:487] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 -> Peer 1e8cadb26fe0428a964f66ef8172f3be (127.14.57.252:33819): Couldn't send request to peer 1e8cadb26fe0428a964f66ef8172f3be. Status: Network error: Client connection negotiation failed: client connection to 127.14.57.252:33819: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260810 08:00:30.373765 15111 tablet_service.cc:1467] Processing CreateTablet for tablet e1b42eac370145d4b684456639968485 (DEFAULT_TABLE table=default.Table_7f3a4e19ded64b3783a00b3d4c6839c3 [id=6c6afcf578534997878b98feb9574e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.408028 15478 ts_tablet_manager.cc:1918] T 35b6da60a5544d7e9b657a5f1e6f90ff P 3fc5e568f3724f158b616c303efb9eae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 08:00:30.421111 14687 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_a6ce85a21a354e7c92f158c1ea3212b0" } modify_external_catalogs: true
I20260810 08:00:30.421772 14687 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_a6ce85a21a354e7c92f158c1ea3212b0" } modify_external_catalogs: true
I20260810 08:00:30.423892 14691 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_b01de34f04234e38bd01d49939e0c2b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:30.425307 14691 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b01de34f04234e38bd01d49939e0c2b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:30.429112 14692 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_d9fa6a06be6e42d1b504e0741c5462be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:30.430683 14692 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9fa6a06be6e42d1b504e0741c5462be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:30.440800 14686 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_cd979630b0c2429a848a3baa03e7e25e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:30.444904 14686 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd979630b0c2429a848a3baa03e7e25e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:30.447489 14695 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_4951d1d0071f4f25afa6ada7bc5840ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:30.449529 14695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4951d1d0071f4f25afa6ada7bc5840ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:30.457379 14960 tablet_service.cc:1514] Processing DeleteTablet for tablet e162a89dcf5a4890aa5a93dfeadd131e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:29 UTC) from {username='slave'} at 127.0.0.1:34630
W20260810 08:00:24.888842 15528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 08:00:24.786222 15531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 08:00:30.468626 14689 catalog_manager.cc:6040] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: default.Table_d81fe7cc77ad45d39c8b44e1299dbefc [id=a705653aa3484a69b369e7e39273d28e] alter complete (version 1)
I20260810 08:00:30.470914 14960 tablet_service.cc:1467] Processing CreateTablet for tablet 0687d323788642f798465ece973702bb (DEFAULT_TABLE table=default.Table_44d94cfdac894ff68cfbb8984912231c [id=0d60b763c4804818ae26fa693a4fd808]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.472024 14960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0687d323788642f798465ece973702bb. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 08:00:30.472368 14690 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.14.57.195:51883 (request call id 38) took 933 ms (client timeout 1000 ms). Trace:
W20260810 08:00:30.472669 14690 rpcz_store.cc:269] 0810 08:00:29.538795 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:29.539106 (+ 311us) service_pool.cc:227] Handling call
0810 08:00:29.584083 (+ 44977us) spinlock_profiling.cc:243] Waited 35.4 ms on lock 0x7b5400021554. stack: 00007fe2372c4033 00007fe2372c3f30 00007fe236e45534 00000000004db2b4 00000000004db249 00007fe23e1d0af0 00007fe23e253f2a 00007fe24011fd6c 00007fe240219c3e 00007fe23dd50fb4 00007fe23dd50f42 00007fe23dd50e71 00007fe23dd50dec 00007fe23dd500a2 00007fe23bffa94c 00007fe23bff9d86
0810 08:00:30.472307 (+888224us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":9,"spinlock_wait_cycles":81350528}
W20260810 08:00:30.476742 15530 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 5680 milliseconds
I20260810 08:00:30.476850 15454 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260810 08:00:30.478865 15454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 08:00:30.377190 15147 raft_consensus.cc:3037] T fbf62f4eab8b49f282df3ab4d95d886f P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:30.490710 14966 tablet_service.cc:1467] Processing CreateTablet for tablet 5f2e51d10e7142938e03b2ef687b0edd (DEFAULT_TABLE table=default.Table_4f0bc756581c45958dc46729a9430f79 [id=87b8720fcdf143299d050c9546dcd91d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.491770 14966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f2e51d10e7142938e03b2ef687b0edd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.530400 15454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 08:00:30.535440 15454 hybrid_clock.cc:648] HybridClock initialized: now 1786348830535381 us; error 75 us; skew 500 ppm
I20260810 08:00:30.537487 15454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 08:00:30.538976 14963 tablet_service.cc:1467] Processing CreateTablet for tablet 2fbda1b3b5b4482ba6f03f3e275d94e7 (DEFAULT_TABLE table=default.Table_fea0f565d8a0450a91e8729141f89d99 [id=3ffa6d723f2344eaac1c5f831d3320a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.540297 14963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fbda1b3b5b4482ba6f03f3e275d94e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.575249 15454 webserver.cc:460] Webserver started at http://127.14.57.252:34789/ using document root <none> and password file <none>
I20260810 08:00:30.576715 15454 fs_manager.cc:359] Metadata directory not provided
I20260810 08:00:30.577092 15454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260810 08:00:30.592658 14918 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.195:37729, user_credentials={real_user=slave}} blocked reactor thread for 106932us
I20260810 08:00:30.602452 14965 tablet_service.cc:1467] Processing CreateTablet for tablet 8a0e1ad5653f46b4861b3e39629fbd2c (DEFAULT_TABLE table=default.Table_3d528c5b4aa54991bf128ada67924546 [id=8c493c9dfd3f4f358bc02f33869f3a16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.603472 14965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a0e1ad5653f46b4861b3e39629fbd2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.611809 15454 fs_manager.cc:709] Time spent opening directory manager: real 0.030s user 0.005s sys 0.008s
I20260810 08:00:30.622570 14964 tablet_service.cc:1467] Processing CreateTablet for tablet 4ca688b4031c41a9884829fbd0be1f33 (DEFAULT_TABLE table=default.Table_357e1765e09e4c4ab104ee0436bc1091 [id=da2bb7febfe9421380f61df1393ea3b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.623692 14964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ca688b4031c41a9884829fbd0be1f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.624290 15716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 08:00:30.625753 15454 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.005s sys 0.001s
I20260810 08:00:30.626166 15454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-2/data,/tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-2/wal
uuid: "1e8cadb26fe0428a964f66ef8172f3be"
format_stamp: "Formatted at 2026-08-10 08:00:12 on dist-test-slave-p3t5"
I20260810 08:00:30.629827 15454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 08:00:30.651176 14690 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_d1964b15134148e1a0a49f4ffe679821" } modify_external_catalogs: true
I20260810 08:00:30.651764 14690 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_d1964b15134148e1a0a49f4ffe679821" } modify_external_catalogs: true
I20260810 08:00:30.654241 14688 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_849976332e234e2bad3d238fc268803f" } new_table_name: "default.Table_c782f8d897974cef9ad4827497661a17" modify_external_catalogs: true
I20260810 08:00:30.675426 14961 tablet_service.cc:1467] Processing CreateTablet for tablet 0a60001d76604d5085c843961d52213f (DEFAULT_TABLE table=default.Table_ebe6adac96b74785b939b2a0d8e7a192 [id=f2c914866c3f418caf5e08a2ad03a91b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.676954 14961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a60001d76604d5085c843961d52213f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.367332 15439 consensus_queue.cc:260] T 25de7bf64bf6450ba17b7748568bc0ed P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:30.421671 15447 raft_consensus.cc:482] T 2d295305b5a843eaa84c8ad666355b84 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:30.696605 14967 tablet_service.cc:1467] Processing CreateTablet for tablet e162a89dcf5a4890aa5a93dfeadd131e (DEFAULT_TABLE table=default.Table_ae0fee60b7704372bf9ac04a22df6876 [id=bdcacd958d574265adebc1fd1cb6c1fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.699697 15447 raft_consensus.cc:504] T 2d295305b5a843eaa84c8ad666355b84 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:30.700603 14967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e162a89dcf5a4890aa5a93dfeadd131e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.701865 15447 leader_election.cc:288] T 2d295305b5a843eaa84c8ad666355b84 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:30.713127 15478 log.cc:1196] T 35b6da60a5544d7e9b657a5f1e6f90ff P 3fc5e568f3724f158b616c303efb9eae: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal/wals/35b6da60a5544d7e9b657a5f1e6f90ff
I20260810 08:00:30.713872 15640 raft_consensus.cc:482] T 37d8a37a16f34a0285c43c0d71b5f072 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:30.715116 15478 ts_tablet_manager.cc:1939] T 35b6da60a5544d7e9b657a5f1e6f90ff P 3fc5e568f3724f158b616c303efb9eae: Deleting consensus metadata
I20260810 08:00:30.726387 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 88) took 1648 ms. Trace:
I20260810 08:00:30.598014 15145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf62f4eab8b49f282df3ab4d95d886f" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:30.613377 15147 raft_consensus.cc:2454] T fbf62f4eab8b49f282df3ab4d95d886f P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 1.
I20260810 08:00:30.618067 15112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b18980bffe6045b5b69bcecb47b4c711. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.732365 15111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1b42eac370145d4b684456639968485. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.734853 14691 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_ebe6adac96b74785b939b2a0d8e7a192" } new_table_name: "default.Table_814fa7cd026940c385ff0c1e0ec55a01" modify_external_catalogs: true
I20260810 08:00:30.626797 15110 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c152aab7e244afd857534869b13815c with delete_type TABLET_DATA_DELETED (replaced by tablet 8a8454f693494ab3b82f5b85c2d62bb2) from {username='slave'} at 127.0.0.1:43002
W20260810 08:00:30.627374 15079 leader_election.cc:341] T d22a778bfe784b2f8a7b4d2216909f10 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:30.739089 15079 leader_election.cc:302] T d22a778bfe784b2f8a7b4d2216909f10 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
W20260810 08:00:30.739858 15079 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.193:43601, user_credentials={real_user=slave}} blocked reactor thread for 112515us
I20260810 08:00:30.740649 15617 raft_consensus.cc:2726] T d22a778bfe784b2f8a7b4d2216909f10 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:30.681071 15479 ts_tablet_manager.cc:1918] T 2cca8a18778c49d0be01363c31ef180b P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:30.741715 15479 log.cc:1196] T 2cca8a18778c49d0be01363c31ef180b P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/2cca8a18778c49d0be01363c31ef180b
I20260810 08:00:30.742360 15479 ts_tablet_manager.cc:1939] T 2cca8a18778c49d0be01363c31ef180b P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:30.744222 15479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43002 (request call id 58) took 2924 ms. Trace:
I20260810 08:00:30.744524 15479 rpcz_store.cc:276] 0810 08:00:27.819573 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:27.846866 (+ 27293us) service_pool.cc:227] Handling call
0810 08:00:30.274950 (+2428084us) ts_tablet_manager.cc:1145] Deleting tablet 2cca8a18778c49d0be01363c31ef180b
0810 08:00:30.275007 (+ 57us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:30.563759 (+288752us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:30.680765 (+117006us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:30.744160 (+ 63395us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2427067}
I20260810 08:00:30.746699 15479 tablet_replica.cc:330] T d22a778bfe784b2f8a7b4d2216909f10 P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:30.747681 15479 raft_consensus.cc:2229] T d22a778bfe784b2f8a7b4d2216909f10 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:30.748178 15479 raft_consensus.cc:2258] T d22a778bfe784b2f8a7b4d2216909f10 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:30.750300 15479 ts_tablet_manager.cc:1905] T d22a778bfe784b2f8a7b4d2216909f10 P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:30.618263 15269 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fae4809ef5e4a9fb16976674e287f03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:28 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:30.619786 15273 tablet_service.cc:1467] Processing CreateTablet for tablet 20a316e404414b288a315a5386208111 (DEFAULT_TABLE table=default.Table_d1964b15134148e1a0a49f4ffe679821 [id=1953381bfd954f849df38454058a6b94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.753773 15478 rpcz_store.cc:276] 0810 08:00:29.078358 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:29.079297 (+ 939us) service_pool.cc:227] Handling call
0810 08:00:30.057876 (+978579us) ts_tablet_manager.cc:1145] Deleting tablet 35b6da60a5544d7e9b657a5f1e6f90ff
0810 08:00:30.057935 (+ 59us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:30.318120 (+260185us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:30.407798 (+ 89678us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:30.726318 (+318520us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":977924}
I20260810 08:00:30.754719 15273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20a316e404414b288a315a5386208111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.755079 15478 tablet_replica.cc:330] T 1461cdeda5664afd9c4c2b45199179b6 P 3fc5e568f3724f158b616c303efb9eae: stopping tablet replica
I20260810 08:00:30.621508 15278 tablet_service.cc:1467] Processing CreateTablet for tablet abc607ab01f74b2c9c693d6dea73090c (DEFAULT_TABLE table=default.Table_849976332e234e2bad3d238fc268803f [id=b864f8b7b9914c35bb9f6de6f767a67f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.756736 15278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abc607ab01f74b2c9c693d6dea73090c. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 08:00:30.394076 15076 leader_election.cc:341] T 2d295305b5a843eaa84c8ad666355b84 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
W20260810 08:00:30.757237 15079 leader_election.cc:341] T 2d295305b5a843eaa84c8ad666355b84 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:30.758054 15079 leader_election.cc:302] T 2d295305b5a843eaa84c8ad666355b84 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:30.761368 15617 raft_consensus.cc:2726] T 2d295305b5a843eaa84c8ad666355b84 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:30.762759 15110 tablet_service.cc:1467] Processing CreateTablet for tablet ae51ddeb22fe4cf7a8aa2d8b2b70a535 (DEFAULT_TABLE table=default.Table_82ef137d777f463c843b60a4dc498741 [id=df7590268b84411d983967b1b05d419f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.763886 15110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae51ddeb22fe4cf7a8aa2d8b2b70a535. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.764285 15478 raft_consensus.cc:2229] T 1461cdeda5664afd9c4c2b45199179b6 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:30.768607 15478 raft_consensus.cc:2258] T 1461cdeda5664afd9c4c2b45199179b6 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:30.773155 15454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 08:00:30.774505 15119 tablet_service.cc:1467] Processing CreateTablet for tablet 461de46896ee492b9f7cd1202208c19a (DEFAULT_TABLE table=default.Table_bfb2fd6d4f974132b165f34aedb323bc [id=7bab6eb9b0c041a28751b184f70e132a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.775586 15454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 08:00:30.775735 15119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 461de46896ee492b9f7cd1202208c19a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.776232 15454 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260810 08:00:30.641330 15277 consensus_meta.cc:287] T 5f2e51d10e7142938e03b2ef687b0edd P 3fc5e568f3724f158b616c303efb9eae: Time spent flushing consensus metadata: real 0.521s user 0.003s sys 0.003s
I20260810 08:00:30.663419 15691 raft_consensus.cc:482] T 37d8a37a16f34a0285c43c0d71b5f072 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:30.782305 15691 raft_consensus.cc:504] T 37d8a37a16f34a0285c43c0d71b5f072 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:30.782866 15113 tablet_service.cc:1467] Processing CreateTablet for tablet fd97cbfb16bf4ee2b4eed1e8bbbc7224 (DEFAULT_TABLE table=default.Table_290c3db6ca0e4fdf8e007f723b11d27d [id=97abcf4bf320496ebbec1fd2ed20765b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.784405 15691 leader_election.cc:288] T 37d8a37a16f34a0285c43c0d71b5f072 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:30.784853 15691 raft_consensus.cc:482] T 25de7bf64bf6450ba17b7748568bc0ed P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:30.785359 15113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd97cbfb16bf4ee2b4eed1e8bbbc7224. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 08:00:30.601069 15276 consensus_meta.cc:287] T 0687d323788642f798465ece973702bb P 3fc5e568f3724f158b616c303efb9eae: Time spent flushing consensus metadata: real 0.570s user 0.006s sys 0.000s
W20260810 08:00:30.689741 15490 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.611s user 0.014s sys 0.004s
W20260810 08:00:30.786774 15490 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.708s user 0.014s sys 0.004s
I20260810 08:00:30.785238 15691 raft_consensus.cc:504] T 25de7bf64bf6450ba17b7748568bc0ed P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:30.789031 15490 tablet_bootstrap.cc:492] T 68bc39707f44457fbf57c217ad827348 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:30.790120 15691 leader_election.cc:288] T 25de7bf64bf6450ba17b7748568bc0ed P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
W20260810 08:00:30.789973 15076 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.195:37729, user_credentials={real_user=slave}} blocked reactor thread for 412778us
I20260810 08:00:30.747143 15439 ts_tablet_manager.cc:1428] T 25de7bf64bf6450ba17b7748568bc0ed P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.389s user 0.010s sys 0.004s
I20260810 08:00:30.792028 15116 tablet_service.cc:1467] Processing CreateTablet for tablet a92d01d83dbf4b6ba47464dcefa8b2ae (DEFAULT_TABLE table=default.Table_9a8eb842ebf244d5abbfb0f43373ca08 [id=23114ccea26d48079f910570c9580a71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.792367 15439 tablet_bootstrap.cc:492] T d22a778bfe784b2f8a7b4d2216909f10 P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:30.793275 15116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a92d01d83dbf4b6ba47464dcefa8b2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.797448 15439 tablet_bootstrap.cc:654] T d22a778bfe784b2f8a7b4d2216909f10 P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
W20260810 08:00:30.798395 14681 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 4381ccb8711744ff8f9766ef49c53e28 (table default.Table_4f3d374da9fd4fcaaff35b3e4c3bdbf7 [id=f07affbf494748a88874b8f1edd3f7b6], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4381ccb8711744ff8f9766ef49c53e28 consensus state has no leader
I20260810 08:00:30.800076 15117 tablet_service.cc:1467] Processing CreateTablet for tablet 33cacf1276d44b24abe6aff12ea8ea51 (DEFAULT_TABLE table=default.Table_d612355f9d634ad88ba3d45cf5ace9ec [id=a7103ecd6fad419aa96ab65ded74ef6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.801211 15117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33cacf1276d44b24abe6aff12ea8ea51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.816965 15482 ts_tablet_manager.cc:1918] T 1461cdeda5664afd9c4c2b45199179b6 P a91b27762182454baada71e91ceab224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 08:00:30.816960 15118 tablet_service.cc:1467] Processing CreateTablet for tablet 20a316e404414b288a315a5386208111 (DEFAULT_TABLE table=default.Table_d1964b15134148e1a0a49f4ffe679821 [id=1953381bfd954f849df38454058a6b94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.819770 15118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20a316e404414b288a315a5386208111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.789489 15490 ts_tablet_manager.cc:1397] T 68bc39707f44457fbf57c217ad827348 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.950s user 0.021s sys 0.005s
W20260810 08:00:30.846077 14689 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.14.57.193:35941 (request call id 34) took 946 ms (client timeout 1000 ms). Trace:
I20260810 08:00:30.856587 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet 2cca8a18778c49d0be01363c31ef180b (table default.Table_1e20f010e84746b0a487c82539f9ab8a [id=82b31a19f2724a95938f0ae5b4700d97]) successfully deleted
I20260810 08:00:30.860536 15482 log.cc:1196] T 1461cdeda5664afd9c4c2b45199179b6 P a91b27762182454baada71e91ceab224: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal/wals/1461cdeda5664afd9c4c2b45199179b6
I20260810 08:00:30.876825 15482 ts_tablet_manager.cc:1939] T 1461cdeda5664afd9c4c2b45199179b6 P a91b27762182454baada71e91ceab224: Deleting consensus metadata
I20260810 08:00:30.877791 15482 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34630 (request call id 86) took 1535 ms. Trace:
I20260810 08:00:30.835996 15145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d295305b5a843eaa84c8ad666355b84" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:30.878343 15145 raft_consensus.cc:2454] T 2d295305b5a843eaa84c8ad666355b84 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 0.
W20260810 08:00:30.748346 15280 consensus_meta.cc:287] T 4ca688b4031c41a9884829fbd0be1f33 P 3fc5e568f3724f158b616c303efb9eae: Time spent flushing consensus metadata: real 0.510s user 0.006s sys 0.001s
I20260810 08:00:30.780387 15478 ts_tablet_manager.cc:1905] T 1461cdeda5664afd9c4c2b45199179b6 P 3fc5e568f3724f158b616c303efb9eae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:30.828191 15490 raft_consensus.cc:348] T 68bc39707f44457fbf57c217ad827348 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:30.879720 15490 raft_consensus.cc:374] T 68bc39707f44457fbf57c217ad827348 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:30.879992 15490 raft_consensus.cc:729] T 68bc39707f44457fbf57c217ad827348 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:30.880678 15490 consensus_queue.cc:260] T 68bc39707f44457fbf57c217ad827348 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:30.857142 15111 tablet_service.cc:1467] Processing CreateTablet for tablet abc607ab01f74b2c9c693d6dea73090c (DEFAULT_TABLE table=default.Table_849976332e234e2bad3d238fc268803f [id=b864f8b7b9914c35bb9f6de6f767a67f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.882614 15111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abc607ab01f74b2c9c693d6dea73090c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.715126 15640 raft_consensus.cc:504] T 37d8a37a16f34a0285c43c0d71b5f072 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:30.884127 15437 tablet_bootstrap.cc:492] T d22a778bfe784b2f8a7b4d2216909f10 P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:30.884601 15437 ts_tablet_manager.cc:1397] T d22a778bfe784b2f8a7b4d2216909f10 P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 1.040s user 0.019s sys 0.005s
I20260810 08:00:30.886812 14959 tablet_service.cc:1467] Processing CreateTablet for tablet 68048e2768cf441d906e675254175c9c (DEFAULT_TABLE table=default.Table_4ff94124f5cb4710a0ad3ff1781efcd3 [id=0b1956596de94b569ebd9f1816e4b699]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.724288 15275 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ddae63b29c34e1397c5765e52e1478f with delete_type TABLET_DATA_DELETED (replaced by tablet 561cbc6a4eb14d3584f88ea26a0fea7a) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:30.747574 15268 tablet_service.cc:1514] Processing DeleteTablet for tablet 114d3eac994f466bb1787e066de05e2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:25 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:30.832880 15267 tablet_service.cc:1467] Processing CreateTablet for tablet 461de46896ee492b9f7cd1202208c19a (DEFAULT_TABLE table=default.Table_bfb2fd6d4f974132b165f34aedb323bc [id=7bab6eb9b0c041a28751b184f70e132a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.893002 15267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 461de46896ee492b9f7cd1202208c19a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.894243 14687 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet cf8bb1c44a7b4d17bddeff8b2fe81b35
I20260810 08:00:30.898839 14690 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet 20a316e404414b288a315a5386208111
I20260810 08:00:30.908128 14957 tablet_service.cc:1467] Processing CreateTablet for tablet 3281dd757ff44d9498ef70ad372eb150 (DEFAULT_TABLE table=default.Table_409addd050ee45b981bddfacfab24c03 [id=76166bd7cb584067b7d2e305603a840b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.910912 15490 ts_tablet_manager.cc:1428] T 68bc39707f44457fbf57c217ad827348 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.089s user 0.006s sys 0.000s
I20260810 08:00:30.838289 15270 tablet_service.cc:1514] Processing DeleteTablet for tablet 2faab63c5023431c98e7b01ce61e04c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:24 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:30.729033 15271 tablet_service.cc:1467] Processing CreateTablet for tablet ae51ddeb22fe4cf7a8aa2d8b2b70a535 (DEFAULT_TABLE table=default.Table_82ef137d777f463c843b60a4dc498741 [id=df7590268b84411d983967b1b05d419f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.912356 15271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae51ddeb22fe4cf7a8aa2d8b2b70a535. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.923714 14693 catalog_manager.cc:5587] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265) does not have the latest schema for tablet 49c35b7f4a72437fa0675abe8a8f3a71 (table default.Table_1567aaa1d2ab44028745e82c29232682 [id=6cf63d3e931a4f6487074494254805bb]). Expected version 4 got 0
I20260810 08:00:30.923306 14956 tablet_service.cc:1467] Processing CreateTablet for tablet 655fd33901974b9c85f52f92ff677247 (DEFAULT_TABLE table=default.Table_69ee062d338749dbba588a79aee54e1e [id=c80b653f9aec49baa511433216b67e29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:30.924997 14956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 655fd33901974b9c85f52f92ff677247. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 08:00:30.881582 14962 consensus_meta.cc:287] T 3855a2b4d90a47ad918c7a90cc8c7e4a P a91b27762182454baada71e91ceab224: Time spent flushing consensus metadata: real 0.561s user 0.010s sys 0.000s
I20260810 08:00:30.927093 14957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3281dd757ff44d9498ef70ad372eb150. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.932730 15478 ts_tablet_manager.cc:1918] T 1461cdeda5664afd9c4c2b45199179b6 P 3fc5e568f3724f158b616c303efb9eae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 08:00:30.933090 15478 log.cc:1196] T 1461cdeda5664afd9c4c2b45199179b6 P 3fc5e568f3724f158b616c303efb9eae: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal/wals/1461cdeda5664afd9c4c2b45199179b6
I20260810 08:00:30.933516 14959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68048e2768cf441d906e675254175c9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:30.933876 15490 tablet_bootstrap.cc:492] T 924c0f73638f48a88c08d45d459132e1 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:30.929706 14693 catalog_manager.cc:5587] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265) does not have the latest schema for tablet 2d295305b5a843eaa84c8ad666355b84 (table default.Table_d5c08115c8b24ad48a9686e48a45ab82 [id=bb6d5450013a45b1bb6a85bc9e4e6d9c]). Expected version 1 got 0
I20260810 08:00:30.941388 14693 catalog_manager.cc:5587] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265) does not have the latest schema for tablet 5a22a48e2ba843b3a32a8900ab937ea0 (table default.Table_8aecb3c6298849c38bcafee965161494 [id=ac3a5787385940bca8030d7bac893a27]). Expected version 3 got 0
I20260810 08:00:30.944674 15675 consensus_queue.cc:1035] T 728f8f80b3bb4b10871d8b400f25377b P a91b27762182454baada71e91ceab224 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 08:00:30.942368 14688 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet abc607ab01f74b2c9c693d6dea73090c consensus state has no leader
W20260810 08:00:30.948230 14691 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0a60001d76604d5085c843961d52213f consensus state has no leader
W20260810 08:00:30.954037 14918 consensus_peers.cc:487] T 2faab63c5023431c98e7b01ce61e04c2 P a91b27762182454baada71e91ceab224 -> Peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Couldn't send request to peer 3fc5e568f3724f158b616c303efb9eae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2faab63c5023431c98e7b01ce61e04c2. This is attempt 1: this message will repeat every 5th retry.
I20260810 08:00:30.962317 15437 raft_consensus.cc:348] T d22a778bfe784b2f8a7b4d2216909f10 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:30.965924 14694 catalog_manager.cc:5587] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729) does not have the latest schema for tablet 5a22a48e2ba843b3a32a8900ab937ea0 (table default.Table_8aecb3c6298849c38bcafee965161494 [id=ac3a5787385940bca8030d7bac893a27]). Expected version 3 got 0
I20260810 08:00:30.967114 14694 catalog_manager.cc:5495] T 49c35b7f4a72437fa0675abe8a8f3a71 P 3fc5e568f3724f158b616c303efb9eae reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fc5e568f3724f158b616c303efb9eae (127.14.57.195). New cstate: current_term: 1 leader_uuid: "3fc5e568f3724f158b616c303efb9eae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } health_report { overall_health: UNKNOWN } } }
I20260810 08:00:30.968693 14678 catalog_manager.cc:4841] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): tablet 35b6da60a5544d7e9b657a5f1e6f90ff (table default.Table_6e428ff2c4c7459b84d2882a089b490d [id=64f8570d723d4f5faba292689c535c01]) successfully deleted
I20260810 08:00:30.969393 14694 catalog_manager.cc:5587] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729) does not have the latest schema for tablet 49c35b7f4a72437fa0675abe8a8f3a71 (table default.Table_1567aaa1d2ab44028745e82c29232682 [id=6cf63d3e931a4f6487074494254805bb]). Expected version 4 got 0
I20260810 08:00:30.970435 14694 catalog_manager.cc:5587] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729) does not have the latest schema for tablet 2d295305b5a843eaa84c8ad666355b84 (table default.Table_d5c08115c8b24ad48a9686e48a45ab82 [id=bb6d5450013a45b1bb6a85bc9e4e6d9c]). Expected version 1 got 0
I20260810 08:00:30.975198 15478 ts_tablet_manager.cc:1939] T 1461cdeda5664afd9c4c2b45199179b6 P 3fc5e568f3724f158b616c303efb9eae: Deleting consensus metadata
I20260810 08:00:30.980700 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 85) took 1904 ms. Trace:
I20260810 08:00:30.994230 15478 rpcz_store.cc:276] 0810 08:00:29.076524 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:29.076653 (+ 129us) service_pool.cc:227] Handling call
0810 08:00:30.754918 (+1678265us) ts_tablet_manager.cc:1145] Deleting tablet 1461cdeda5664afd9c4c2b45199179b6
0810 08:00:30.754982 (+ 64us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:30.886401 (+131419us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:30.932496 (+ 46095us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:30.980614 (+ 48118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4989,"tablet-delete.queue_time_us":1674141}
I20260810 08:00:30.999874 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 91) took 1297 ms. Trace:
I20260810 08:00:31.000501 15478 rpcz_store.cc:276] 0810 08:00:29.702180 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:29.707493 (+ 5313us) service_pool.cc:227] Handling call
0810 08:00:30.999467 (+1291974us) ts_tablet_manager.cc:1145] Deleting tablet 8364045e3fa2441e8dc1a1d87f73361d
0810 08:00:30.999515 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:30.999816 (+ 301us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1290743}
I20260810 08:00:31.002437 15490 tablet_bootstrap.cc:654] T 924c0f73638f48a88c08d45d459132e1 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:31.013978 15302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37d8a37a16f34a0285c43c0d71b5f072" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:31.018565 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 92) took 1026 ms. Trace:
I20260810 08:00:31.018910 15302 raft_consensus.cc:2454] T 37d8a37a16f34a0285c43c0d71b5f072 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 0.
I20260810 08:00:31.019001 15478 rpcz_store.cc:276] 0810 08:00:29.992035 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:29.992178 (+ 143us) service_pool.cc:227] Handling call
0810 08:00:31.018052 (+1025874us) ts_tablet_manager.cc:1145] Deleting tablet 3ac554bd37b045a2a8cf2743f4645f28
0810 08:00:31.018113 (+ 61us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:31.018494 (+ 381us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1025264}
I20260810 08:00:31.020551 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 93) took 1028 ms. Trace:
I20260810 08:00:31.022698 15490 tablet_bootstrap.cc:492] T 924c0f73638f48a88c08d45d459132e1 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:31.023116 15490 ts_tablet_manager.cc:1397] T 924c0f73638f48a88c08d45d459132e1 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.111s user 0.011s sys 0.004s
I20260810 08:00:31.028012 15490 raft_consensus.cc:348] T 924c0f73638f48a88c08d45d459132e1 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:31.030073 15490 raft_consensus.cc:374] T 924c0f73638f48a88c08d45d459132e1 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:31.030328 15490 raft_consensus.cc:729] T 924c0f73638f48a88c08d45d459132e1 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:31.031957 15145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25de7bf64bf6450ba17b7748568bc0ed" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:31.032608 15145 raft_consensus.cc:2454] T 25de7bf64bf6450ba17b7748568bc0ed P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 0.
W20260810 08:00:31.040805 14693 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.14.57.194:55587 (request call id 43) took 849 ms (client timeout 1000 ms). Trace:
I20260810 08:00:31.041982 15490 consensus_queue.cc:260] T 924c0f73638f48a88c08d45d459132e1 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:31.020933 15478 rpcz_store.cc:276] 0810 08:00:29.992364 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:29.992911 (+ 547us) service_pool.cc:227] Handling call
0810 08:00:31.020104 (+1027193us) ts_tablet_manager.cc:1145] Deleting tablet e162a89dcf5a4890aa5a93dfeadd131e
0810 08:00:31.020158 (+ 54us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:31.020488 (+ 330us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1026501}
I20260810 08:00:31.047344 15478 tablet_replica.cc:330] T 0fae4809ef5e4a9fb16976674e287f03 P 3fc5e568f3724f158b616c303efb9eae: stopping tablet replica
I20260810 08:00:31.050042 15437 raft_consensus.cc:374] T d22a778bfe784b2f8a7b4d2216909f10 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:31.050341 15437 raft_consensus.cc:729] T d22a778bfe784b2f8a7b4d2216909f10 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:31.050952 15437 consensus_queue.cc:260] T d22a778bfe784b2f8a7b4d2216909f10 P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:31.051537 15490 ts_tablet_manager.cc:1428] T 924c0f73638f48a88c08d45d459132e1 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.028s user 0.003s sys 0.002s
I20260810 08:00:30.990023 15640 leader_election.cc:288] T 37d8a37a16f34a0285c43c0d71b5f072 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:30.991721 14955 tablet_service.cc:1467] Processing CreateTablet for tablet b18980bffe6045b5b69bcecb47b4c711 (DEFAULT_TABLE table=default.Table_64e1c230e9624fc9ba6c328346150ffe [id=46e6c2115d8c4b5e9907ed76b273638a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:31.052590 14955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b18980bffe6045b5b69bcecb47b4c711. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:31.018307 14954 tablet_service.cc:1514] Processing DeleteTablet for tablet a1c4f8f514c94128aedd07bd5efdc3be with delete_type TABLET_DATA_DELETED (replaced by tablet a4a78b2a960d4d78aa1f48a9c9edf3be) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:31.022570 14953 tablet_service.cc:1514] Processing DeleteTablet for tablet 2faab63c5023431c98e7b01ce61e04c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:24 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:31.027607 14986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d295305b5a843eaa84c8ad666355b84" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:31.058616 15490 tablet_bootstrap.cc:492] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:31.065820 15490 tablet_bootstrap.cc:654] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:31.069712 14952 tablet_service.cc:1514] Processing DeleteTablet for tablet 114d3eac994f466bb1787e066de05e2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:25 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:31.071002 14952 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fae4809ef5e4a9fb16976674e287f03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:28 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:31.072010 14952 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c152aab7e244afd857534869b13815c with delete_type TABLET_DATA_DELETED (replaced by tablet 8a8454f693494ab3b82f5b85c2d62bb2) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:31.075013 15478 raft_consensus.cc:2229] T 0fae4809ef5e4a9fb16976674e287f03 P 3fc5e568f3724f158b616c303efb9eae [term 1 LEADER]: Raft consensus shutting down.
W20260810 08:00:31.075572 14693 rpcz_store.cc:269] 0810 08:00:30.191239 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:30.191434 (+ 195us) service_pool.cc:227] Handling call
0810 08:00:30.456401 (+264967us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b54000213a8. stack: 00007fe2372c4033 00007fe2372c3f30 00007fe236e45534 00000000004db2b4 00000000004db249 00000000004db0e1 00000000004da6bc 00007fe24011fd54 00007fe240219c3e 00007fe23dd50fb4 00007fe23dd50f42 00007fe23dd50e71 00007fe23dd50dec 00007fe23dd500a2 00007fe23bffa94c 00007fe23bff9d86
0810 08:00:30.651417 (+195016us) spinlock_profiling.cc:243] Waited 194 ms on lock 0x7b5400021554. stack: 00007fe2372c4033 00007fe2372c3f30 00007fe236e45534 00000000004db2b4 00000000004db249 00007fe23e1d0af0 00007fe23e253f2a 00007fe24011fd6c 00007fe240219c3e 00007fe23dd50fb4 00007fe23dd50f42 00007fe23dd50e71 00007fe23dd50dec 00007fe23dd500a2 00007fe23bffa94c 00007fe23bff9d86
0810 08:00:31.040709 (+389292us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":11,"spinlock_wait_cycles":709549568}
I20260810 08:00:31.077968 14952 tablet_service.cc:1514] Processing DeleteTablet for tablet 728f8f80b3bb4b10871d8b400f25377b with delete_type TABLET_DATA_DELETED (replaced by tablet e13b91d61063446f9d1454da4f8bb706) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:31.079495 15478 pending_rounds.cc:62] T 0fae4809ef5e4a9fb16976674e287f03 P 3fc5e568f3724f158b616c303efb9eae: Trying to abort 1 pending ops.
I20260810 08:00:31.082914 15478 pending_rounds.cc:68] T 0fae4809ef5e4a9fb16976674e287f03 P 3fc5e568f3724f158b616c303efb9eae: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316884805002457088 op_type: NO_OP noop_request { }
I20260810 08:00:31.085592 15723 raft_consensus.cc:482] T d0761b090d8147b8a5537b0ed51108a9 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:31.090742 15723 raft_consensus.cc:504] T d0761b090d8147b8a5537b0ed51108a9 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:31.093741 15723 leader_election.cc:288] T d0761b090d8147b8a5537b0ed51108a9 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
W20260810 08:00:30.846591 14689 rpcz_store.cc:269] 0810 08:00:29.899868 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:29.900002 (+ 134us) service_pool.cc:227] Handling call
0810 08:00:30.846003 (+946001us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 08:00:30.416162 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 08:00:30.416568 (+ 406us) write_op.cc:432] Acquiring schema lock in shared mode
0810 08:00:30.416593 (+ 25us) write_op.cc:435] Acquired schema lock
0810 08:00:30.416605 (+ 12us) tablet.cc:660] Decoding operations
0810 08:00:30.416900 (+ 295us) write_op.cc:620] Acquiring the partition lock for write op
0810 08:00:30.416959 (+ 59us) write_op.cc:641] Partition lock acquired for write op
0810 08:00:30.416987 (+ 28us) tablet.cc:683] Acquiring locks for 1 operations
0810 08:00:30.417171 (+ 184us) tablet.cc:699] Row locks acquired
0810 08:00:30.417185 (+ 14us) write_op.cc:260] PREPARE: finished
0810 08:00:30.417346 (+ 161us) write_op.cc:270] Start()
0810 08:00:30.417488 (+ 142us) write_op.cc:276] Timestamp: P: 1786348830417248 usec, L: 0
0810 08:00:30.417506 (+ 18us) op_driver.cc:331] REPLICATION: starting
0810 08:00:30.418184 (+ 678us) log.cc:842] Serialized 356 byte log entry
0810 08:00:30.432033 (+ 13849us) op_driver.cc:444] REPLICATION: finished
0810 08:00:30.432523 (+ 490us) write_op.cc:301] APPLY: starting
0810 08:00:30.432579 (+ 56us) tablet.cc:1365] starting BulkCheckPresence
0810 08:00:30.432701 (+ 122us) tablet.cc:1368] finished BulkCheckPresence
0810 08:00:30.432848 (+ 147us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 08:00:30.433102 (+ 254us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 08:00:30.433271 (+ 169us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 08:00:30.433316 (+ 45us) write_op.cc:312] APPLY: finished
0810 08:00:30.433521 (+ 205us) log.cc:842] Serialized 23 byte log entry
0810 08:00:30.433711 (+ 190us) write_op.cc:489] Releasing partition, row and schema locks
0810 08:00:30.433776 (+ 65us) write_op.cc:454] Released schema lock
0810 08:00:30.433809 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0810 08:00:30.434048 (+ 239us) write_op.cc:489] Releasing partition, row and schema locks
0810 08:00:30.434075 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":10,"child_traces":[["op",{"apply.queue_time_us":276,"apply.run_cpu_time_us":1752,"apply.run_wall_time_us":1752,"num_ops":1,"prepare.queue_time_us":125,"prepare.run_cpu_time_us":2417,"prepare.run_wall_time_us":2418,"raft.queue_time_us":2638,"raft.run_cpu_time_us":657,"raft.run_wall_time_us":658,"replication_time_us":14432}]]}
I20260810 08:00:31.095611 14985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37d8a37a16f34a0285c43c0d71b5f072" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:31.096241 14985 raft_consensus.cc:2454] T 37d8a37a16f34a0285c43c0d71b5f072 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 0.
I20260810 08:00:31.038316 14965 tablet_service.cc:1467] Processing CreateTablet for tablet a4a78b2a960d4d78aa1f48a9c9edf3be (DEFAULT_TABLE table=default.Table_89fe7ad723964551909b73ff6b72b9af [id=ce96e7fb7c46404a859ca08c1fd1b26a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:31.099483 14965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4a78b2a960d4d78aa1f48a9c9edf3be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:31.094245 15478 raft_consensus.cc:2866] T 0fae4809ef5e4a9fb16976674e287f03 P 3fc5e568f3724f158b616c303efb9eae [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260810 08:00:31.099918 15478 raft_consensus.cc:2258] T 0fae4809ef5e4a9fb16976674e287f03 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:31.101807 15478 ts_tablet_manager.cc:1905] T 0fae4809ef5e4a9fb16976674e287f03 P 3fc5e568f3724f158b616c303efb9eae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:31.045239 14964 tablet_service.cc:1467] Processing CreateTablet for tablet e1b42eac370145d4b684456639968485 (DEFAULT_TABLE table=default.Table_7f3a4e19ded64b3783a00b3d4c6839c3 [id=6c6afcf578534997878b98feb9574e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:31.126830 14964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1b42eac370145d4b684456639968485. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:31.135037 14952 tablet_service.cc:1467] Processing CreateTablet for tablet ae51ddeb22fe4cf7a8aa2d8b2b70a535 (DEFAULT_TABLE table=default.Table_82ef137d777f463c843b60a4dc498741 [id=df7590268b84411d983967b1b05d419f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:31.136332 14952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae51ddeb22fe4cf7a8aa2d8b2b70a535. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:31.148584 15122 tablet_service.cc:1514] Processing DeleteTablet for tablet cf8bb1c44a7b4d17bddeff8b2fe81b35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:30 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:31.161978 14918 leader_election.cc:302] T 37d8a37a16f34a0285c43c0d71b5f072 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224; no voters:
I20260810 08:00:31.163938 15675 raft_consensus.cc:2781] T 37d8a37a16f34a0285c43c0d71b5f072 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:31.164271 15675 raft_consensus.cc:482] T 37d8a37a16f34a0285c43c0d71b5f072 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:31.164535 15675 raft_consensus.cc:3037] T 37d8a37a16f34a0285c43c0d71b5f072 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:31.175048 15675 raft_consensus.cc:504] T 37d8a37a16f34a0285c43c0d71b5f072 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:31.176971 15675 leader_election.cc:288] T 37d8a37a16f34a0285c43c0d71b5f072 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Requested vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:30.878319 15482 rpcz_store.cc:276] 0810 08:00:29.341961 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:29.354791 (+ 12830us) service_pool.cc:227] Handling call
0810 08:00:29.703664 (+348873us) ts_tablet_manager.cc:1145] Deleting tablet 1461cdeda5664afd9c4c2b45199179b6
0810 08:00:29.703705 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:30.276828 (+573123us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:30.816699 (+539871us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:30.877733 (+ 61034us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41470,"tablet-delete.queue_time_us":249741}
I20260810 08:00:31.196353 15478 ts_tablet_manager.cc:1918] T 0fae4809ef5e4a9fb16976674e287f03 P 3fc5e568f3724f158b616c303efb9eae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 08:00:31.196959 15478 log.cc:1196] T 0fae4809ef5e4a9fb16976674e287f03 P 3fc5e568f3724f158b616c303efb9eae: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal/wals/0fae4809ef5e4a9fb16976674e287f03
I20260810 08:00:31.197747 15482 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34630 (request call id 89) took 1402 ms. Trace:
I20260810 08:00:31.198096 15478 ts_tablet_manager.cc:1939] T 0fae4809ef5e4a9fb16976674e287f03 P 3fc5e568f3724f158b616c303efb9eae: Deleting consensus metadata
I20260810 08:00:31.198248 15482 rpcz_store.cc:276] 0810 08:00:29.795173 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:29.795300 (+ 127us) service_pool.cc:227] Handling call
0810 08:00:31.197324 (+1402024us) ts_tablet_manager.cc:1145] Deleting tablet 8364045e3fa2441e8dc1a1d87f73361d
0810 08:00:31.197372 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:31.197670 (+ 298us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1401148}
I20260810 08:00:31.200546 15482 tablet_replica.cc:330] T 2faab63c5023431c98e7b01ce61e04c2 P a91b27762182454baada71e91ceab224: stopping tablet replica
I20260810 08:00:31.115258 15234 leader_election.cc:302] T fbf62f4eab8b49f282df3ab4d95d886f P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, 9d7219686f304f62a54acc7497cbbd07; no voters:
I20260810 08:00:31.205339 15723 raft_consensus.cc:2781] T fbf62f4eab8b49f282df3ab4d95d886f P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Leader election won for term 1
I20260810 08:00:31.205965 15723 raft_consensus.cc:686] T fbf62f4eab8b49f282df3ab4d95d886f P 3fc5e568f3724f158b616c303efb9eae [term 1 LEADER]: Becoming Leader. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Running, Role: LEADER
I20260810 08:00:31.206982 15723 consensus_queue.cc:237] T fbf62f4eab8b49f282df3ab4d95d886f P 3fc5e568f3724f158b616c303efb9eae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
W20260810 08:00:31.211908 15234 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.194:36265, user_credentials={real_user=slave}} blocked reactor thread for 96705.8us
I20260810 08:00:31.215975 15234 leader_election.cc:302] T 2d295305b5a843eaa84c8ad666355b84 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, 9d7219686f304f62a54acc7497cbbd07; no voters:
W20260810 08:00:31.218230 15234 leader_election.cc:341] T 37d8a37a16f34a0285c43c0d71b5f072 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Not found: Tablet not found: 37d8a37a16f34a0285c43c0d71b5f072
I20260810 08:00:31.219045 15691 raft_consensus.cc:2781] T 2d295305b5a843eaa84c8ad666355b84 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:31.223388 15691 raft_consensus.cc:482] T 2d295305b5a843eaa84c8ad666355b84 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:31.223711 15691 raft_consensus.cc:3037] T 2d295305b5a843eaa84c8ad666355b84 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:31.223701 15234 leader_election.cc:302] T 25de7bf64bf6450ba17b7748568bc0ed P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, 9d7219686f304f62a54acc7497cbbd07; no voters:
I20260810 08:00:31.234050 15723 raft_consensus.cc:2781] T 25de7bf64bf6450ba17b7748568bc0ed P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:31.234644 15723 raft_consensus.cc:482] T 25de7bf64bf6450ba17b7748568bc0ed P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:31.235136 15723 raft_consensus.cc:3037] T 25de7bf64bf6450ba17b7748568bc0ed P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:31.236580 15691 raft_consensus.cc:504] T 2d295305b5a843eaa84c8ad666355b84 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:31.240141 15691 leader_election.cc:288] T 2d295305b5a843eaa84c8ad666355b84 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Requested vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:31.254207 15112 tablet_service.cc:1514] Processing DeleteTablet for tablet 20a316e404414b288a315a5386208111 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:30 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:31.273435 15490 tablet_bootstrap.cc:492] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:31.273919 15490 ts_tablet_manager.cc:1397] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.215s user 0.025s sys 0.034s
I20260810 08:00:31.276194 15490 raft_consensus.cc:348] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:31.276865 15490 raft_consensus.cc:374] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:31.277134 15490 raft_consensus.cc:729] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:31.252367 15439 tablet_bootstrap.cc:492] T d22a778bfe784b2f8a7b4d2216909f10 P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:31.277899 15439 ts_tablet_manager.cc:1397] T d22a778bfe784b2f8a7b4d2216909f10 P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.486s user 0.005s sys 0.005s
I20260810 08:00:31.280622 15439 raft_consensus.cc:348] T d22a778bfe784b2f8a7b4d2216909f10 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:31.289899 15490 consensus_queue.cc:260] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:31.305281 15112 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cca8a18778c49d0be01363c31ef180b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:26 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:31.307564 15490 ts_tablet_manager.cc:1428] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.033s user 0.005s sys 0.000s
I20260810 08:00:31.324827 15490 tablet_bootstrap.cc:492] T 4381ccb8711744ff8f9766ef49c53e28 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:31.216106 14953 tablet_service.cc:1467] Processing CreateTablet for tablet abc607ab01f74b2c9c693d6dea73090c (DEFAULT_TABLE table=default.Table_849976332e234e2bad3d238fc268803f [id=b864f8b7b9914c35bb9f6de6f767a67f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:31.218150 14954 tablet_service.cc:1467] Processing CreateTablet for tablet 461de46896ee492b9f7cd1202208c19a (DEFAULT_TABLE table=default.Table_bfb2fd6d4f974132b165f34aedb323bc [id=7bab6eb9b0c041a28751b184f70e132a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:31.219866 14966 tablet_service.cc:1514] Processing DeleteTablet for tablet fbf62f4eab8b49f282df3ab4d95d886f with delete_type TABLET_DATA_DELETED (replaced by tablet e1b42eac370145d4b684456639968485) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:31.220662 14962 tablet_service.cc:1514] Processing DeleteTablet for tablet 37d8a37a16f34a0285c43c0d71b5f072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:25 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:31.222503 14960 tablet_service.cc:1467] Processing CreateTablet for tablet a92d01d83dbf4b6ba47464dcefa8b2ae (DEFAULT_TABLE table=default.Table_9a8eb842ebf244d5abbfb0f43373ca08 [id=23114ccea26d48079f910570c9580a71]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 08:00:31.245792 14918 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.57.195:37729, user_credentials={real_user=slave}} blocked reactor thread for 82042.6us
W20260810 08:00:31.254817 14963 consensus_meta.cc:287] T 2fbda1b3b5b4482ba6f03f3e275d94e7 P a91b27762182454baada71e91ceab224: Time spent flushing consensus metadata: real 0.610s user 0.006s sys 0.000s
I20260810 08:00:31.256079 15437 ts_tablet_manager.cc:1428] T d22a778bfe784b2f8a7b4d2216909f10 P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.371s user 0.006s sys 0.000s
I20260810 08:00:31.365543 15750 tablet.cc:1722] T 5a22a48e2ba843b3a32a8900ab937ea0 P 9d7219686f304f62a54acc7497cbbd07: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260810 08:00:31.370747 15490 tablet_bootstrap.cc:654] T 4381ccb8711744ff8f9766ef49c53e28 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:31.376263 15749 raft_consensus.cc:482] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:31.282169 14967 tablet_service.cc:1467] Processing CreateTablet for tablet fd97cbfb16bf4ee2b4eed1e8bbbc7224 (DEFAULT_TABLE table=default.Table_290c3db6ca0e4fdf8e007f723b11d27d [id=97abcf4bf320496ebbec1fd2ed20765b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:31.307396 14951 tablet_service.cc:1467] Processing CreateTablet for tablet 33cacf1276d44b24abe6aff12ea8ea51 (DEFAULT_TABLE table=default.Table_d612355f9d634ad88ba3d45cf5ace9ec [id=a7103ecd6fad419aa96ab65ded74ef6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:31.308872 14950 tablet_service.cc:1514] Processing DeleteTablet for tablet cf8bb1c44a7b4d17bddeff8b2fe81b35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:30 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:31.309911 14949 tablet_service.cc:1514] Processing DeleteTablet for tablet 20a316e404414b288a315a5386208111 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:30 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:31.377354 15749 raft_consensus.cc:504] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:31.324919 14961 tablet_service.cc:1467] Processing CreateTablet for tablet 20a316e404414b288a315a5386208111 (DEFAULT_TABLE table=default.Table_d1964b15134148e1a0a49f4ffe679821 [id=1953381bfd954f849df38454058a6b94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:31.384753 15748 raft_consensus.cc:482] T 68bc39707f44457fbf57c217ad827348 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:31.385319 15748 raft_consensus.cc:504] T 68bc39707f44457fbf57c217ad827348 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:31.388826 15749 leader_election.cc:288] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), a91b27762182454baada71e91ceab224 (127.14.57.193:43601)
I20260810 08:00:31.389863 15748 leader_election.cc:288] T 68bc39707f44457fbf57c217ad827348 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), a91b27762182454baada71e91ceab224 (127.14.57.193:43601)
I20260810 08:00:31.260713 15739 consensus_queue.cc:1035] T 114d3eac994f466bb1787e066de05e2c P 3fc5e568f3724f158b616c303efb9eae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 08:00:31.393961 15439 raft_consensus.cc:374] T d22a778bfe784b2f8a7b4d2216909f10 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:31.264528 15723 raft_consensus.cc:504] T 25de7bf64bf6450ba17b7748568bc0ed P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:31.394589 15439 raft_consensus.cc:729] T d22a778bfe784b2f8a7b4d2216909f10 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:31.396368 15723 leader_election.cc:288] T 25de7bf64bf6450ba17b7748568bc0ed P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Requested vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:31.290992 15691 consensus_queue.cc:1035] T 49c35b7f4a72437fa0675abe8a8f3a71 P 3fc5e568f3724f158b616c303efb9eae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 08:00:31.337299 15478 tablet_replica.cc:330] T 114d3eac994f466bb1787e066de05e2c P 3fc5e568f3724f158b616c303efb9eae: stopping tablet replica
I20260810 08:00:31.401087 15478 raft_consensus.cc:2229] T 114d3eac994f466bb1787e066de05e2c P 3fc5e568f3724f158b616c303efb9eae [term 1 LEADER]: Raft consensus shutting down.
I20260810 08:00:31.401934 15478 pending_rounds.cc:62] T 114d3eac994f466bb1787e066de05e2c P 3fc5e568f3724f158b616c303efb9eae: Trying to abort 1 pending ops.
I20260810 08:00:31.402211 15478 pending_rounds.cc:68] T 114d3eac994f466bb1787e066de05e2c P 3fc5e568f3724f158b616c303efb9eae: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316884804675788800 op_type: NO_OP noop_request { }
I20260810 08:00:31.402666 15478 raft_consensus.cc:2866] T 114d3eac994f466bb1787e066de05e2c P 3fc5e568f3724f158b616c303efb9eae [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260810 08:00:31.403030 15478 raft_consensus.cc:2258] T 114d3eac994f466bb1787e066de05e2c P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:31.362561 15300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37d8a37a16f34a0285c43c0d71b5f072" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae"
W20260810 08:00:31.260124 15234 leader_election.cc:341] T 2d295305b5a843eaa84c8ad666355b84 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
W20260810 08:00:31.412571 15234 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.193:43601, user_credentials={real_user=slave}} blocked reactor thread for 152470us
I20260810 08:00:31.414664 15234 leader_election.cc:302] T 37d8a37a16f34a0285c43c0d71b5f072 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224; no voters: 9d7219686f304f62a54acc7497cbbd07
I20260810 08:00:31.410970 15478 ts_tablet_manager.cc:1905] T 114d3eac994f466bb1787e066de05e2c P 3fc5e568f3724f158b616c303efb9eae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:31.404561 15300 raft_consensus.cc:3037] T 37d8a37a16f34a0285c43c0d71b5f072 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:31.426913 15270 tablet_service.cc:1467] Processing CreateTablet for tablet a92d01d83dbf4b6ba47464dcefa8b2ae (DEFAULT_TABLE table=default.Table_9a8eb842ebf244d5abbfb0f43373ca08 [id=23114ccea26d48079f910570c9580a71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:31.428043 15270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a92d01d83dbf4b6ba47464dcefa8b2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:31.430724 15268 tablet_service.cc:1467] Processing CreateTablet for tablet 33cacf1276d44b24abe6aff12ea8ea51 (DEFAULT_TABLE table=default.Table_d612355f9d634ad88ba3d45cf5ace9ec [id=a7103ecd6fad419aa96ab65ded74ef6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:31.432220 15268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33cacf1276d44b24abe6aff12ea8ea51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:31.441861 14985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8bb1c44a7b4d17bddeff8b2fe81b35" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:31.441284 14986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68bc39707f44457fbf57c217ad827348" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
W20260810 08:00:31.443558 15079 leader_election.cc:341] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
W20260810 08:00:31.444667 15079 leader_election.cc:341] T 68bc39707f44457fbf57c217ad827348 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:31.445318 15275 tablet_service.cc:1514] Processing DeleteTablet for tablet cf8bb1c44a7b4d17bddeff8b2fe81b35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:30 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:31.445791 14988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25de7bf64bf6450ba17b7748568bc0ed" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:31.449373 15275 tablet_service.cc:1514] Processing DeleteTablet for tablet 20a316e404414b288a315a5386208111 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:30 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:31.448220 15267 tablet_service.cc:1467] Processing CreateTablet for tablet fd97cbfb16bf4ee2b4eed1e8bbbc7224 (DEFAULT_TABLE table=default.Table_290c3db6ca0e4fdf8e007f723b11d27d [id=97abcf4bf320496ebbec1fd2ed20765b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:31.456601 15275 tablet_service.cc:1514] Processing DeleteTablet for tablet 35b6da60a5544d7e9b657a5f1e6f90ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:26 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:31.458988 15267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd97cbfb16bf4ee2b4eed1e8bbbc7224. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:31.460835 15755 tablet.cc:1722] T 5a22a48e2ba843b3a32a8900ab937ea0 P a91b27762182454baada71e91ceab224: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260810 08:00:31.462052 15280 tablet_service.cc:1514] Processing DeleteTablet for tablet 37d8a37a16f34a0285c43c0d71b5f072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:25 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:31.466111 15275 tablet_service.cc:1514] Processing DeleteTablet for tablet fbf62f4eab8b49f282df3ab4d95d886f with delete_type TABLET_DATA_DELETED (replaced by tablet e1b42eac370145d4b684456639968485) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:31.476892 15278 tablet_service.cc:1514] Processing DeleteTablet for tablet 728f8f80b3bb4b10871d8b400f25377b with delete_type TABLET_DATA_DELETED (replaced by tablet e13b91d61063446f9d1454da4f8bb706) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:31.477900 15490 tablet_bootstrap.cc:492] T 4381ccb8711744ff8f9766ef49c53e28 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:31.478396 15490 ts_tablet_manager.cc:1397] T 4381ccb8711744ff8f9766ef49c53e28 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.154s user 0.007s sys 0.004s
I20260810 08:00:31.480717 15490 raft_consensus.cc:348] T 4381ccb8711744ff8f9766ef49c53e28 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:31.481371 15490 raft_consensus.cc:374] T 4381ccb8711744ff8f9766ef49c53e28 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:31.481666 15490 raft_consensus.cc:729] T 4381ccb8711744ff8f9766ef49c53e28 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:31.482331 15490 consensus_queue.cc:260] T 4381ccb8711744ff8f9766ef49c53e28 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:31.482759 15439 consensus_queue.cc:260] T d22a778bfe784b2f8a7b4d2216909f10 P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:31.485127 15490 ts_tablet_manager.cc:1428] T 4381ccb8711744ff8f9766ef49c53e28 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 08:00:31.486123 15490 tablet_bootstrap.cc:492] T b5a0e0af742340b2945be2102768b307 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:31.491777 15490 tablet_bootstrap.cc:654] T b5a0e0af742340b2945be2102768b307 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:31.492396 15748 raft_consensus.cc:482] T 924c0f73638f48a88c08d45d459132e1 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:31.492815 15748 raft_consensus.cc:504] T 924c0f73638f48a88c08d45d459132e1 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:31.494971 15748 leader_election.cc:288] T 924c0f73638f48a88c08d45d459132e1 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), a91b27762182454baada71e91ceab224 (127.14.57.193:43601)
I20260810 08:00:31.504585 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "924c0f73638f48a88c08d45d459132e1" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:31.505343 14951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33cacf1276d44b24abe6aff12ea8ea51. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 08:00:31.506116 15079 leader_election.cc:341] T 924c0f73638f48a88c08d45d459132e1 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:31.513902 14954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 461de46896ee492b9f7cd1202208c19a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:31.518119 15490 tablet_bootstrap.cc:492] T b5a0e0af742340b2945be2102768b307 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:31.518548 15490 ts_tablet_manager.cc:1397] T b5a0e0af742340b2945be2102768b307 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.005s
I20260810 08:00:31.519441 14953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abc607ab01f74b2c9c693d6dea73090c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:31.520673 15490 raft_consensus.cc:348] T b5a0e0af742340b2945be2102768b307 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:31.521308 15490 raft_consensus.cc:374] T b5a0e0af742340b2945be2102768b307 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:31.521559 15490 raft_consensus.cc:729] T b5a0e0af742340b2945be2102768b307 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:31.522315 15490 consensus_queue.cc:260] T b5a0e0af742340b2945be2102768b307 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:31.535051 15490 ts_tablet_manager.cc:1428] T b5a0e0af742340b2945be2102768b307 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.016s user 0.005s sys 0.001s
I20260810 08:00:31.535720 15750 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:43002 (request call id 100) took 1299 ms. Trace:
I20260810 08:00:31.535976 15490 tablet_bootstrap.cc:492] T cb37a3a87f8b48a68c19510f306073c2 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:31.537983 15437 tablet_bootstrap.cc:492] T 2cca8a18778c49d0be01363c31ef180b P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:31.536204 15750 rpcz_store.cc:276] 0810 08:00:30.236790 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:30.386535 (+149745us) service_pool.cc:227] Handling call
0810 08:00:31.535642 (+1149107us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 08:00:30.592009 (+ 0us) alter_schema_op.cc:102] PREPARE ALTER-SCHEMA: Starting
0810 08:00:30.592374 (+ 365us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0810 08:00:30.592403 (+ 29us) alter_schema_op.cc:65] Acquired schema lock
0810 08:00:30.592471 (+ 68us) alter_schema_op.cc:117] PREPARE ALTER-SCHEMA: finished
0810 08:00:30.592590 (+ 119us) alter_schema_op.cc:126] START. Timestamp: 1786348830592488
0810 08:00:30.592620 (+ 30us) op_driver.cc:331] REPLICATION: starting
0810 08:00:30.593168 (+ 548us) log.cc:842] Serialized 173 byte log entry
0810 08:00:31.363807 (+770639us) op_driver.cc:444] REPLICATION: finished
0810 08:00:31.365248 (+ 1441us) alter_schema_op.cc:131] APPLY ALTER-SCHEMA: Starting
0810 08:00:31.373003 (+ 7755us) log.cc:842] Serialized 5 byte log entry
0810 08:00:31.509438 (+136435us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:31.534855 (+ 25417us) log.cc:842] Serialized 14 byte log entry
0810 08:00:31.535486 (+ 631us) alter_schema_op.cc:71] Released schema lock
0810 08:00:31.535512 (+ 26us) alter_schema_op.cc:184] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1209,"prepare.queue_time_us":13374,"prepare.run_cpu_time_us":8809,"prepare.run_wall_time_us":16425,"raft.queue_time_us":14433,"raft.run_cpu_time_us":1588,"raft.run_wall_time_us":1589,"replication_time_us":771098,"thread_start_us":23651,"threads_started":3}]]}
I20260810 08:00:31.558604 15482 raft_consensus.cc:2229] T 2faab63c5023431c98e7b01ce61e04c2 P a91b27762182454baada71e91ceab224 [term 1 LEADER]: Raft consensus shutting down.
I20260810 08:00:31.559548 15482 raft_consensus.cc:2258] T 2faab63c5023431c98e7b01ce61e04c2 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:31.561896 14960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a92d01d83dbf4b6ba47464dcefa8b2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:31.550292 15490 tablet_bootstrap.cc:654] T cb37a3a87f8b48a68c19510f306073c2 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:31.566931 14967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd97cbfb16bf4ee2b4eed1e8bbbc7224. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:31.569293 14961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20a316e404414b288a315a5386208111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:31.579664 15437 tablet_bootstrap.cc:654] T 2cca8a18778c49d0be01363c31ef180b P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:31.584275 15490 tablet_bootstrap.cc:492] T cb37a3a87f8b48a68c19510f306073c2 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:31.585706 15490 ts_tablet_manager.cc:1397] T cb37a3a87f8b48a68c19510f306073c2 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.050s user 0.009s sys 0.008s
I20260810 08:00:31.587266 15482 ts_tablet_manager.cc:1905] T 2faab63c5023431c98e7b01ce61e04c2 P a91b27762182454baada71e91ceab224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:31.588356 15490 raft_consensus.cc:348] T cb37a3a87f8b48a68c19510f306073c2 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:31.589088 15490 raft_consensus.cc:374] T cb37a3a87f8b48a68c19510f306073c2 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:31.589322 15490 raft_consensus.cc:729] T cb37a3a87f8b48a68c19510f306073c2 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:31.589932 15490 consensus_queue.cc:260] T cb37a3a87f8b48a68c19510f306073c2 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:31.594719 15490 ts_tablet_manager.cc:1428] T cb37a3a87f8b48a68c19510f306073c2 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260810 08:00:31.597069 15490 tablet_bootstrap.cc:492] T 1f427f4093e5417a9ca79b1805b0675e P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:31.607703 14689 catalog_manager.cc:5587] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601) does not have the latest schema for tablet 5a22a48e2ba843b3a32a8900ab937ea0 (table default.Table_8aecb3c6298849c38bcafee965161494 [id=ac3a5787385940bca8030d7bac893a27]). Expected version 3 got 0
I20260810 08:00:31.608409 15490 tablet_bootstrap.cc:654] T 1f427f4093e5417a9ca79b1805b0675e P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:31.609297 14689 catalog_manager.cc:5587] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601) does not have the latest schema for tablet 49c35b7f4a72437fa0675abe8a8f3a71 (table default.Table_1567aaa1d2ab44028745e82c29232682 [id=6cf63d3e931a4f6487074494254805bb]). Expected version 4 got 0
I20260810 08:00:31.612649 15750 tablet.cc:1717] T 5a22a48e2ba843b3a32a8900ab937ea0 P 9d7219686f304f62a54acc7497cbbd07: Skipping requested alter to schema version 3, tablet already version 3
I20260810 08:00:31.618010 15755 tablet.cc:1717] T 5a22a48e2ba843b3a32a8900ab937ea0 P a91b27762182454baada71e91ceab224: Skipping requested alter to schema version 3, tablet already version 3
I20260810 08:00:31.623426 15490 tablet_bootstrap.cc:492] T 1f427f4093e5417a9ca79b1805b0675e P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:31.623885 15490 ts_tablet_manager.cc:1397] T 1f427f4093e5417a9ca79b1805b0675e P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.002s
I20260810 08:00:31.624320 14693 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_62105ef471f3450bb3aabefb32baa96d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:31.624550 14694 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_6cbd44cdd6a44bc4b3eb991d874353eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:31.626047 14693 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62105ef471f3450bb3aabefb32baa96d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:31.626113 15490 raft_consensus.cc:348] T 1f427f4093e5417a9ca79b1805b0675e P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:31.626113 14691 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_27c2e400be3e44cd8d3f1738490182e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:31.626718 15490 raft_consensus.cc:374] T 1f427f4093e5417a9ca79b1805b0675e P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:31.626981 15490 raft_consensus.cc:729] T 1f427f4093e5417a9ca79b1805b0675e P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
W20260810 08:00:31.627514 14691 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_27c2e400be3e44cd8d3f1738490182e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:31.627537 15490 consensus_queue.cc:260] T 1f427f4093e5417a9ca79b1805b0675e P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:31.628468 14692 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_357e1765e09e4c4ab104ee0436bc1091" } new_table_name: "default.Table_f8637789a86d4e0da9392352f395f0c3" modify_external_catalogs: true
I20260810 08:00:31.630096 14690 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_409addd050ee45b981bddfacfab24c03" } modify_external_catalogs: true
I20260810 08:00:31.630717 14690 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_409addd050ee45b981bddfacfab24c03" } modify_external_catalogs: true
I20260810 08:00:31.634146 14687 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_4f3d374da9fd4fcaaff35b3e4c3bdbf7" } modify_external_catalogs: true
I20260810 08:00:31.634902 14687 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_4f3d374da9fd4fcaaff35b3e4c3bdbf7" } modify_external_catalogs: true
I20260810 08:00:31.636338 15479 ts_tablet_manager.cc:1918] T d22a778bfe784b2f8a7b4d2216909f10 P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:31.629495 15490 ts_tablet_manager.cc:1428] T 1f427f4093e5417a9ca79b1805b0675e P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260810 08:00:31.637526 15490 tablet_bootstrap.cc:492] T 561cbc6a4eb14d3584f88ea26a0fea7a P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:31.637842 14686 master_service.cc:946] ReplaceTablet: received request to replace tablet adada21bc1bf4f19a6347d0935bd74bd from {username='slave'} at 127.0.0.1:42300
I20260810 08:00:31.642023 14695 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_64bd369e459e44d9ac85bf7defb3c86c" } new_table_name: "default.Table_02bc67e213c241fabead65c8dc785aa8" modify_external_catalogs: true
I20260810 08:00:31.642829 15490 tablet_bootstrap.cc:654] T 561cbc6a4eb14d3584f88ea26a0fea7a P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:31.629341 14688 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_d34e1afecbcf418483dec42d2f76f8d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:31.659142 15490 tablet_bootstrap.cc:492] T 561cbc6a4eb14d3584f88ea26a0fea7a P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
W20260810 08:00:31.659277 14688 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d34e1afecbcf418483dec42d2f76f8d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:31.659541 15490 ts_tablet_manager.cc:1397] T 561cbc6a4eb14d3584f88ea26a0fea7a P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.000s
I20260810 08:00:31.661687 15479 log.cc:1196] T d22a778bfe784b2f8a7b4d2216909f10 P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/d22a778bfe784b2f8a7b4d2216909f10
I20260810 08:00:31.661870 15490 raft_consensus.cc:348] T 561cbc6a4eb14d3584f88ea26a0fea7a P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:31.662400 15479 ts_tablet_manager.cc:1939] T d22a778bfe784b2f8a7b4d2216909f10 P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
W20260810 08:00:31.662366 14694 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6cbd44cdd6a44bc4b3eb991d874353eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:31.662525 15490 raft_consensus.cc:374] T 561cbc6a4eb14d3584f88ea26a0fea7a P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:31.662758 15490 raft_consensus.cc:729] T 561cbc6a4eb14d3584f88ea26a0fea7a P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:31.663360 15490 consensus_queue.cc:260] T 561cbc6a4eb14d3584f88ea26a0fea7a P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:31.665220 15490 ts_tablet_manager.cc:1428] T 561cbc6a4eb14d3584f88ea26a0fea7a P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
W20260810 08:00:31.672762 14692 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4ca688b4031c41a9884829fbd0be1f33 consensus state has no leader
I20260810 08:00:31.672942 15490 tablet_bootstrap.cc:492] T a244ad3bda5d45f9b89f68b734338d21 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:31.678133 15490 tablet_bootstrap.cc:654] T a244ad3bda5d45f9b89f68b734338d21 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
W20260810 08:00:31.681948 14952 consensus_meta.cc:287] T ae51ddeb22fe4cf7a8aa2d8b2b70a535 P a91b27762182454baada71e91ceab224: Time spent flushing consensus metadata: real 0.537s user 0.009s sys 0.000s
I20260810 08:00:31.685042 14922 leader_election.cc:302] T fbf62f4eab8b49f282df3ab4d95d886f P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a91b27762182454baada71e91ceab224; no voters: 3fc5e568f3724f158b616c303efb9eae, 9d7219686f304f62a54acc7497cbbd07
I20260810 08:00:31.685904 15675 raft_consensus.cc:2726] T fbf62f4eab8b49f282df3ab4d95d886f P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:31.689227 15617 raft_consensus.cc:482] T cb37a3a87f8b48a68c19510f306073c2 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:31.689733 15617 raft_consensus.cc:504] T cb37a3a87f8b48a68c19510f306073c2 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:31.691826 15617 leader_election.cc:288] T cb37a3a87f8b48a68c19510f306073c2 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), a91b27762182454baada71e91ceab224 (127.14.57.193:43601)
I20260810 08:00:31.692967 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb37a3a87f8b48a68c19510f306073c2" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
W20260810 08:00:31.694403 15079 leader_election.cc:341] T cb37a3a87f8b48a68c19510f306073c2 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:31.698184 15112 tablet_service.cc:1467] Processing CreateTablet for tablet 8b5a629a7f8041179a006fa412fb6b48 (DEFAULT_TABLE table=default.Table_d9fa6a06be6e42d1b504e0741c5462be [id=b333dde37a3e456c9f5b361e5125c8cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:31.699287 15112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b5a629a7f8041179a006fa412fb6b48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:31.699493 15278 tablet_service.cc:1514] Processing DeleteTablet for tablet d0761b090d8147b8a5537b0ed51108a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:27 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:31.703337 15490 tablet_bootstrap.cc:492] T a244ad3bda5d45f9b89f68b734338d21 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:31.703733 15490 ts_tablet_manager.cc:1397] T a244ad3bda5d45f9b89f68b734338d21 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.001s
I20260810 08:00:31.704401 15439 ts_tablet_manager.cc:1428] T d22a778bfe784b2f8a7b4d2216909f10 P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.426s user 0.004s sys 0.002s
W20260810 08:00:31.705001 15271 consensus_meta.cc:287] T ae51ddeb22fe4cf7a8aa2d8b2b70a535 P 3fc5e568f3724f158b616c303efb9eae: Time spent flushing consensus metadata: real 0.784s user 0.005s sys 0.001s
I20260810 08:00:31.705364 15439 tablet_bootstrap.cc:492] T 2cca8a18778c49d0be01363c31ef180b P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:31.709285 15490 raft_consensus.cc:348] T a244ad3bda5d45f9b89f68b734338d21 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:31.709997 15490 raft_consensus.cc:374] T a244ad3bda5d45f9b89f68b734338d21 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:31.710247 15490 raft_consensus.cc:729] T a244ad3bda5d45f9b89f68b734338d21 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:31.710857 15490 consensus_queue.cc:260] T a244ad3bda5d45f9b89f68b734338d21 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:31.711627 15617 raft_consensus.cc:482] T a244ad3bda5d45f9b89f68b734338d21 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:31.712033 15617 raft_consensus.cc:504] T a244ad3bda5d45f9b89f68b734338d21 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:31.712793 15490 ts_tablet_manager.cc:1428] T a244ad3bda5d45f9b89f68b734338d21 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20260810 08:00:31.713655 15490 tablet_bootstrap.cc:492] T 8364045e3fa2441e8dc1a1d87f73361d P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:31.715166 15617 leader_election.cc:288] T a244ad3bda5d45f9b89f68b734338d21 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), a91b27762182454baada71e91ceab224 (127.14.57.193:43601)
I20260810 08:00:31.717046 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a244ad3bda5d45f9b89f68b734338d21" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
W20260810 08:00:31.718688 15079 leader_election.cc:341] T a244ad3bda5d45f9b89f68b734338d21 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:31.728020 15490 tablet_bootstrap.cc:654] T 8364045e3fa2441e8dc1a1d87f73361d P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
W20260810 08:00:31.734501 14695 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 579bac622ddf40bf80d01d66ae2fa100 consensus state has no leader
W20260810 08:00:31.738451 14922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.194:36265, user_credentials={real_user=slave}} blocked reactor thread for 53502.7us
I20260810 08:00:31.739084 15490 tablet_bootstrap.cc:492] T 8364045e3fa2441e8dc1a1d87f73361d P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:31.739485 15490 ts_tablet_manager.cc:1397] T 8364045e3fa2441e8dc1a1d87f73361d P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.000s
W20260810 08:00:31.740103 14922 consensus_peers.cc:487] T 728f8f80b3bb4b10871d8b400f25377b P a91b27762182454baada71e91ceab224 -> Peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Couldn't send request to peer 9d7219686f304f62a54acc7497cbbd07. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 728f8f80b3bb4b10871d8b400f25377b. This is attempt 1: this message will repeat every 5th retry.
W20260810 08:00:31.741379 14922 leader_election.cc:341] T 37d8a37a16f34a0285c43c0d71b5f072 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Not found: Tablet not found: 37d8a37a16f34a0285c43c0d71b5f072
I20260810 08:00:31.741699 15490 raft_consensus.cc:348] T 8364045e3fa2441e8dc1a1d87f73361d P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
W20260810 08:00:31.742184 14922 leader_election.cc:341] T 37d8a37a16f34a0285c43c0d71b5f072 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Not found: Tablet not found: 37d8a37a16f34a0285c43c0d71b5f072
I20260810 08:00:31.742367 15490 raft_consensus.cc:374] T 8364045e3fa2441e8dc1a1d87f73361d P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:31.742606 15490 raft_consensus.cc:729] T 8364045e3fa2441e8dc1a1d87f73361d P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:31.748510 15490 consensus_queue.cc:260] T 8364045e3fa2441e8dc1a1d87f73361d P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:31.750568 15490 ts_tablet_manager.cc:1428] T 8364045e3fa2441e8dc1a1d87f73361d P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260810 08:00:31.751423 15490 tablet_bootstrap.cc:492] T e0652a7684b04a89840049ce0d781199 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:31.763355 15490 tablet_bootstrap.cc:654] T e0652a7684b04a89840049ce0d781199 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:31.775854 15439 tablet_bootstrap.cc:654] T 2cca8a18778c49d0be01363c31ef180b P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:31.785475 15479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43002 (request call id 64) took 3907 ms. Trace:
I20260810 08:00:31.785957 15479 rpcz_store.cc:276] 0810 08:00:27.878231 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:27.878426 (+ 195us) service_pool.cc:227] Handling call
0810 08:00:30.746556 (+2868130us) ts_tablet_manager.cc:1145] Deleting tablet d22a778bfe784b2f8a7b4d2216909f10
0810 08:00:30.746597 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:31.247857 (+501260us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:31.636118 (+388261us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:31.785398 (+149280us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2867070}
I20260810 08:00:31.787331 15479 tablet_replica.cc:330] T fbf62f4eab8b49f282df3ab4d95d886f P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:31.788254 15490 tablet_bootstrap.cc:492] T e0652a7684b04a89840049ce0d781199 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:31.788722 15490 ts_tablet_manager.cc:1397] T e0652a7684b04a89840049ce0d781199 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.006s
I20260810 08:00:31.791031 15490 raft_consensus.cc:348] T e0652a7684b04a89840049ce0d781199 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:31.791663 15490 raft_consensus.cc:374] T e0652a7684b04a89840049ce0d781199 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:31.792735 14678 catalog_manager.cc:4841] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): tablet 1461cdeda5664afd9c4c2b45199179b6 (table default.Table_d7227a2f586b4fd29babdc0ae25a0328 [id=5170908091794c4aa1d533a4a0f2b776]) successfully deleted
I20260810 08:00:31.794406 15146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d295305b5a843eaa84c8ad666355b84" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07"
I20260810 08:00:31.794680 15145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0761b090d8147b8a5537b0ed51108a9" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
W20260810 08:00:31.796284 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet 8364045e3fa2441e8dc1a1d87f73361d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8364045e3fa2441e8dc1a1d87f73361d already in progress: creating tablet
W20260810 08:00:31.797204 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet 3ac554bd37b045a2a8cf2743f4645f28 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ac554bd37b045a2a8cf2743f4645f28 already in progress: creating tablet
I20260810 08:00:31.787578 15273 tablet_service.cc:1467] Processing CreateTablet for tablet a59c73c8a95f4230946357b8aa58a1b3 (DEFAULT_TABLE table=default.Table_cd979630b0c2429a848a3baa03e7e25e [id=7b115b827ed74aa5b1f00b4153941bc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:31.790108 15300 raft_consensus.cc:2454] T 37d8a37a16f34a0285c43c0d71b5f072 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 1.
W20260810 08:00:31.798245 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet e162a89dcf5a4890aa5a93dfeadd131e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e162a89dcf5a4890aa5a93dfeadd131e already in progress: creating tablet
I20260810 08:00:31.799361 14678 catalog_manager.cc:4841] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): tablet 0fae4809ef5e4a9fb16976674e287f03 (table default.Table_beb88f8abaa447bfb6962df5718b7313 [id=41bcbdf2ef674d8b8db9284b4b0dcdb9]) successfully deleted
W20260810 08:00:31.800158 14678 catalog_manager.cc:4808] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet 9ddae63b29c34e1397c5765e52e1478f because the tablet was not found. No further retry: Not found: Tablet not found: 9ddae63b29c34e1397c5765e52e1478f
W20260810 08:00:31.801321 14680 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 579bac622ddf40bf80d01d66ae2fa100 (table default.Table_02bc67e213c241fabead65c8dc785aa8 [id=650bc0c97e7842798d575199f1ea662c], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 579bac622ddf40bf80d01d66ae2fa100 consensus state has no leader
I20260810 08:00:31.791905 15490 raft_consensus.cc:729] T e0652a7684b04a89840049ce0d781199 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:31.802011 15145 raft_consensus.cc:2454] T d0761b090d8147b8a5537b0ed51108a9 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 0.
I20260810 08:00:31.802304 15490 consensus_queue.cc:260] T e0652a7684b04a89840049ce0d781199 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:31.802865 15146 raft_consensus.cc:3037] T 2d295305b5a843eaa84c8ad666355b84 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:31.804397 15490 ts_tablet_manager.cc:1428] T e0652a7684b04a89840049ce0d781199 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20260810 08:00:31.804952 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0761b090d8147b8a5537b0ed51108a9" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:31.806913 15144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25de7bf64bf6450ba17b7748568bc0ed" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07"
I20260810 08:00:31.807642 15144 raft_consensus.cc:3037] T 25de7bf64bf6450ba17b7748568bc0ed P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:31.807437 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d295305b5a843eaa84c8ad666355b84" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224"
I20260810 08:00:31.814924 15146 raft_consensus.cc:2454] T 2d295305b5a843eaa84c8ad666355b84 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 1.
I20260810 08:00:31.816566 15273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a59c73c8a95f4230946357b8aa58a1b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:31.817127 15278 tablet_service.cc:1467] Processing CreateTablet for tablet 8b5a629a7f8041179a006fa412fb6b48 (DEFAULT_TABLE table=default.Table_d9fa6a06be6e42d1b504e0741c5462be [id=b333dde37a3e456c9f5b361e5125c8cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:31.818503 15278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b5a629a7f8041179a006fa412fb6b48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:31.822554 15275 tablet_service.cc:1467] Processing CreateTablet for tablet 3593a8e38e1a44f68a54bd52b19a3c6d (DEFAULT_TABLE table=default.Table_b01de34f04234e38bd01d49939e0c2b1 [id=f16f9b3b556646d2bb1c73f026d0a0ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:31.823905 15275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3593a8e38e1a44f68a54bd52b19a3c6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:31.827126 15280 tablet_service.cc:1467] Processing CreateTablet for tablet 3090226feb724bb1aecb0f559d8282a8 (DEFAULT_TABLE table=default.Table_4951d1d0071f4f25afa6ada7bc5840ec [id=df3350a1d74143aa96caec8af21b7e1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:31.788211 15479 raft_consensus.cc:2229] T fbf62f4eab8b49f282df3ab4d95d886f P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:31.828116 15490 tablet_bootstrap.cc:492] T adada21bc1bf4f19a6347d0935bd74bd P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:31.828233 15280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3090226feb724bb1aecb0f559d8282a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:31.828353 15479 raft_consensus.cc:2258] T fbf62f4eab8b49f282df3ab4d95d886f P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 08:00:31.829126 14922 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.Ping -> {remote=127.14.57.252:33819, user_credentials={real_user=slave}} blocked reactor thread for 86433.6us
I20260810 08:00:31.813910 15723 raft_consensus.cc:2741] T 37d8a37a16f34a0285c43c0d71b5f072 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260810 08:00:31.833349 15490 tablet_bootstrap.cc:654] T adada21bc1bf4f19a6347d0935bd74bd P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:31.842434 15437 tablet_bootstrap.cc:492] T 2cca8a18778c49d0be01363c31ef180b P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:31.842774 15271 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40692 (request call id 113) took 1222 ms. Trace:
I20260810 08:00:31.842880 15437 ts_tablet_manager.cc:1397] T 2cca8a18778c49d0be01363c31ef180b P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.305s user 0.007s sys 0.003s
I20260810 08:00:31.843218 15271 rpcz_store.cc:276] 0810 08:00:30.620508 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:30.728098 (+107590us) service_pool.cc:227] Handling call
0810 08:00:30.911894 (+183796us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 08:00:30.911984 (+ 90us) ts_tablet_manager.cc:676] Creating new metadata...
0810 08:00:30.920168 (+ 8184us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:31.705469 (+785301us) tablet_replica.cc:205] Creating consensus instance
0810 08:00:31.842700 (+137231us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 08:00:31.844379 15479 ts_tablet_manager.cc:1905] T fbf62f4eab8b49f282df3ab4d95d886f P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:31.851768 15490 tablet_bootstrap.cc:492] T adada21bc1bf4f19a6347d0935bd74bd P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:31.852240 15490 ts_tablet_manager.cc:1397] T adada21bc1bf4f19a6347d0935bd74bd P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.003s
I20260810 08:00:31.852886 14964 tablet_service.cc:1467] Processing CreateTablet for tablet 8b5a629a7f8041179a006fa412fb6b48 (DEFAULT_TABLE table=default.Table_d9fa6a06be6e42d1b504e0741c5462be [id=b333dde37a3e456c9f5b361e5125c8cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:31.853976 14964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b5a629a7f8041179a006fa412fb6b48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:31.855160 15490 raft_consensus.cc:348] T adada21bc1bf4f19a6347d0935bd74bd P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:31.858268 15490 raft_consensus.cc:374] T adada21bc1bf4f19a6347d0935bd74bd P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:31.858522 15490 raft_consensus.cc:729] T adada21bc1bf4f19a6347d0935bd74bd P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:31.860805 15490 consensus_queue.cc:260] T adada21bc1bf4f19a6347d0935bd74bd P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:31.863075 15490 ts_tablet_manager.cc:1428] T adada21bc1bf4f19a6347d0935bd74bd P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260810 08:00:31.863456 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8bb1c44a7b4d17bddeff8b2fe81b35" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:31.867691 15490 tablet_bootstrap.cc:492] T 5990e7ea243843bb869de5dde3e6cd23 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:31.871220 15144 raft_consensus.cc:2454] T 25de7bf64bf6450ba17b7748568bc0ed P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 1.
I20260810 08:00:31.879582 15490 tablet_bootstrap.cc:654] T 5990e7ea243843bb869de5dde3e6cd23 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:31.857163 15437 raft_consensus.cc:348] T 2cca8a18778c49d0be01363c31ef180b P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:31.867054 14960 tablet_service.cc:1467] Processing CreateTablet for tablet 3593a8e38e1a44f68a54bd52b19a3c6d (DEFAULT_TABLE table=default.Table_b01de34f04234e38bd01d49939e0c2b1 [id=f16f9b3b556646d2bb1c73f026d0a0ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:31.894196 15675 raft_consensus.cc:482] T d22a778bfe784b2f8a7b4d2216909f10 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:31.876691 14988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25de7bf64bf6450ba17b7748568bc0ed" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224"
I20260810 08:00:31.897171 15437 raft_consensus.cc:374] T 2cca8a18778c49d0be01363c31ef180b P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:31.897476 15437 raft_consensus.cc:729] T 2cca8a18778c49d0be01363c31ef180b P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:31.897439 15675 raft_consensus.cc:504] T d22a778bfe784b2f8a7b4d2216909f10 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:31.898268 15437 consensus_queue.cc:260] T 2cca8a18778c49d0be01363c31ef180b P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:31.899406 15675 leader_election.cc:288] T d22a778bfe784b2f8a7b4d2216909f10 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:31.901575 14960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3593a8e38e1a44f68a54bd52b19a3c6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:31.905258 15300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "924c0f73638f48a88c08d45d459132e1" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:31.912853 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68bc39707f44457fbf57c217ad827348" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
W20260810 08:00:31.920706 15234 leader_election.cc:341] T 25de7bf64bf6450ba17b7748568bc0ed P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
W20260810 08:00:31.915128 14922 leader_election.cc:341] T d22a778bfe784b2f8a7b4d2216909f10 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Not found: Tablet not found: d22a778bfe784b2f8a7b4d2216909f10
I20260810 08:00:31.927161 15298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d22a778bfe784b2f8a7b4d2216909f10" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:31.927891 15298 raft_consensus.cc:2454] T d22a778bfe784b2f8a7b4d2216909f10 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 0.
I20260810 08:00:31.928983 15299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a244ad3bda5d45f9b89f68b734338d21" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:31.940181 15302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb37a3a87f8b48a68c19510f306073c2" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:31.949654 15482 ts_tablet_manager.cc:1918] T 2faab63c5023431c98e7b01ce61e04c2 P a91b27762182454baada71e91ceab224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 08:00:31.951803 15482 log.cc:1196] T 2faab63c5023431c98e7b01ce61e04c2 P a91b27762182454baada71e91ceab224: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal/wals/2faab63c5023431c98e7b01ce61e04c2
I20260810 08:00:31.954515 15482 ts_tablet_manager.cc:1939] T 2faab63c5023431c98e7b01ce61e04c2 P a91b27762182454baada71e91ceab224: Deleting consensus metadata
I20260810 08:00:31.956297 14686 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet adada21bc1bf4f19a6347d0935bd74bd
I20260810 08:00:31.958158 15479 ts_tablet_manager.cc:1918] T fbf62f4eab8b49f282df3ab4d95d886f P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:31.959889 15479 log.cc:1196] T fbf62f4eab8b49f282df3ab4d95d886f P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/fbf62f4eab8b49f282df3ab4d95d886f
I20260810 08:00:31.960683 15479 ts_tablet_manager.cc:1939] T fbf62f4eab8b49f282df3ab4d95d886f P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:31.962785 15479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43002 (request call id 65) took 4082 ms. Trace:
I20260810 08:00:31.963433 15617 raft_consensus.cc:482] T b5a0e0af742340b2945be2102768b307 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:31.963940 15617 raft_consensus.cc:504] T b5a0e0af742340b2945be2102768b307 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:31.965780 15617 leader_election.cc:288] T b5a0e0af742340b2945be2102768b307 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:31.963142 15479 rpcz_store.cc:276] 0810 08:00:27.880169 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:27.894701 (+ 14532us) service_pool.cc:227] Handling call
0810 08:00:31.787165 (+3892464us) ts_tablet_manager.cc:1145] Deleting tablet fbf62f4eab8b49f282df3ab4d95d886f
0810 08:00:31.787211 (+ 46us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:31.937488 (+150277us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:31.957902 (+ 20414us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:31.962727 (+ 4825us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3891581}
I20260810 08:00:31.968289 15479 tablet_replica.cc:330] T d0761b090d8147b8a5537b0ed51108a9 P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:31.969465 15479 raft_consensus.cc:2229] T d0761b090d8147b8a5537b0ed51108a9 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:31.970156 15479 raft_consensus.cc:2258] T d0761b090d8147b8a5537b0ed51108a9 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:31.971880 15479 ts_tablet_manager.cc:1905] T d0761b090d8147b8a5537b0ed51108a9 P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:31.979764 15482 tablet_replica.cc:330] T 114d3eac994f466bb1787e066de05e2c P a91b27762182454baada71e91ceab224: stopping tablet replica
I20260810 08:00:32.002789 14918 leader_election.cc:302] T 37d8a37a16f34a0285c43c0d71b5f072 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224; no voters: 9d7219686f304f62a54acc7497cbbd07
I20260810 08:00:32.004007 15675 raft_consensus.cc:2781] T 37d8a37a16f34a0285c43c0d71b5f072 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 08:00:32.004528 15675 raft_consensus.cc:686] T 37d8a37a16f34a0285c43c0d71b5f072 P a91b27762182454baada71e91ceab224 [term 1 LEADER]: Becoming Leader. State: Replica: a91b27762182454baada71e91ceab224, State: Running, Role: LEADER
I20260810 08:00:32.005440 15675 consensus_queue.cc:237] T 37d8a37a16f34a0285c43c0d71b5f072 P a91b27762182454baada71e91ceab224 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.013025 15479 ts_tablet_manager.cc:1918] T d0761b090d8147b8a5537b0ed51108a9 P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:32.014778 15479 log.cc:1196] T d0761b090d8147b8a5537b0ed51108a9 P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/d0761b090d8147b8a5537b0ed51108a9
I20260810 08:00:32.016198 15479 ts_tablet_manager.cc:1939] T d0761b090d8147b8a5537b0ed51108a9 P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:32.029099 15479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43002 (request call id 66) took 3813 ms. Trace:
I20260810 08:00:32.029574 15479 rpcz_store.cc:276] 0810 08:00:28.216078 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:28.246562 (+ 30484us) service_pool.cc:227] Handling call
0810 08:00:31.968025 (+3721463us) ts_tablet_manager.cc:1145] Deleting tablet d0761b090d8147b8a5537b0ed51108a9
0810 08:00:31.968069 (+ 44us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:31.993966 (+ 25897us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:32.012763 (+ 18797us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:32.029025 (+ 16262us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3720806}
I20260810 08:00:32.032517 15479 tablet_replica.cc:330] T cb37a3a87f8b48a68c19510f306073c2 P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:32.034829 15479 raft_consensus.cc:2229] T cb37a3a87f8b48a68c19510f306073c2 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:32.041445 15479 raft_consensus.cc:2258] T cb37a3a87f8b48a68c19510f306073c2 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:32.045358 15479 ts_tablet_manager.cc:1905] T cb37a3a87f8b48a68c19510f306073c2 P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:32.048126 14687 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet 4381ccb8711744ff8f9766ef49c53e28
I20260810 08:00:32.064908 15617 raft_consensus.cc:482] T 4381ccb8711744ff8f9766ef49c53e28 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:32.065349 15617 raft_consensus.cc:504] T 4381ccb8711744ff8f9766ef49c53e28 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.067580 15617 leader_election.cc:288] T 4381ccb8711744ff8f9766ef49c53e28 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), a91b27762182454baada71e91ceab224 (127.14.57.193:43601)
I20260810 08:00:32.092000 15112 tablet_service.cc:1467] Processing CreateTablet for tablet 3593a8e38e1a44f68a54bd52b19a3c6d (DEFAULT_TABLE table=default.Table_b01de34f04234e38bd01d49939e0c2b1 [id=f16f9b3b556646d2bb1c73f026d0a0ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:32.092940 15479 ts_tablet_manager.cc:1918] T cb37a3a87f8b48a68c19510f306073c2 P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:32.093138 15112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3593a8e38e1a44f68a54bd52b19a3c6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:32.093349 15479 log.cc:1196] T cb37a3a87f8b48a68c19510f306073c2 P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/cb37a3a87f8b48a68c19510f306073c2
I20260810 08:00:32.096669 15482 raft_consensus.cc:2229] T 114d3eac994f466bb1787e066de05e2c P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:32.098541 15482 pending_rounds.cc:62] T 114d3eac994f466bb1787e066de05e2c P a91b27762182454baada71e91ceab224: Trying to abort 1 pending ops.
I20260810 08:00:32.098752 15482 pending_rounds.cc:68] T 114d3eac994f466bb1787e066de05e2c P a91b27762182454baada71e91ceab224: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316884804675788800 op_type: NO_OP noop_request { }
I20260810 08:00:32.099076 15482 raft_consensus.cc:2866] T 114d3eac994f466bb1787e066de05e2c P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260810 08:00:32.099375 15482 raft_consensus.cc:2258] T 114d3eac994f466bb1787e066de05e2c P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 08:00:32.099855 14918 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.195:37729, user_credentials={real_user=slave}} blocked reactor thread for 97191.2us
I20260810 08:00:32.100682 14918 leader_election.cc:302] T d22a778bfe784b2f8a7b4d2216909f10 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224; no voters: 9d7219686f304f62a54acc7497cbbd07
I20260810 08:00:32.103215 14686 catalog_manager.cc:6565] ReplaceTablet: tablet adada21bc1bf4f19a6347d0935bd74bd deleted and replaced by tablet d25928e4c39d45e69c3efeb36f03aece
I20260810 08:00:32.113149 15781 raft_consensus.cc:2781] T d22a778bfe784b2f8a7b4d2216909f10 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:32.113575 15781 raft_consensus.cc:482] T d22a778bfe784b2f8a7b4d2216909f10 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:32.113718 15479 ts_tablet_manager.cc:1939] T cb37a3a87f8b48a68c19510f306073c2 P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:32.113960 15781 raft_consensus.cc:3037] T d22a778bfe784b2f8a7b4d2216909f10 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:32.114992 15479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43002 (request call id 67) took 3866 ms. Trace:
I20260810 08:00:32.115300 15479 rpcz_store.cc:276] 0810 08:00:28.248723 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:28.248833 (+ 110us) service_pool.cc:227] Handling call
0810 08:00:32.032373 (+3783540us) ts_tablet_manager.cc:1145] Deleting tablet cb37a3a87f8b48a68c19510f306073c2
0810 08:00:32.032415 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:32.063848 (+ 31433us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:32.092676 (+ 28828us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:32.114932 (+ 22256us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3782910}
I20260810 08:00:32.121851 14960 tablet_service.cc:1467] Processing CreateTablet for tablet 3090226feb724bb1aecb0f559d8282a8 (DEFAULT_TABLE table=default.Table_4951d1d0071f4f25afa6ada7bc5840ec [id=df3350a1d74143aa96caec8af21b7e1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:32.123142 14960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3090226feb724bb1aecb0f559d8282a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:32.126004 15479 tablet_replica.cc:330] T 0fae4809ef5e4a9fb16976674e287f03 P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:32.126731 15479 raft_consensus.cc:2229] T 0fae4809ef5e4a9fb16976674e287f03 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:32.127187 15479 raft_consensus.cc:2258] T 0fae4809ef5e4a9fb16976674e287f03 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:32.128798 15479 ts_tablet_manager.cc:1905] T 0fae4809ef5e4a9fb16976674e287f03 P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:32.135251 15122 tablet_service.cc:1467] Processing CreateTablet for tablet 3090226feb724bb1aecb0f559d8282a8 (DEFAULT_TABLE table=default.Table_4951d1d0071f4f25afa6ada7bc5840ec [id=df3350a1d74143aa96caec8af21b7e1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:32.135828 14964 tablet_service.cc:1467] Processing CreateTablet for tablet a59c73c8a95f4230946357b8aa58a1b3 (DEFAULT_TABLE table=default.Table_cd979630b0c2429a848a3baa03e7e25e [id=7b115b827ed74aa5b1f00b4153941bc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:32.136374 15122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3090226feb724bb1aecb0f559d8282a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:32.136777 14964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a59c73c8a95f4230946357b8aa58a1b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:32.148178 15111 tablet_service.cc:1467] Processing CreateTablet for tablet a59c73c8a95f4230946357b8aa58a1b3 (DEFAULT_TABLE table=default.Table_cd979630b0c2429a848a3baa03e7e25e [id=7b115b827ed74aa5b1f00b4153941bc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:32.149338 15111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a59c73c8a95f4230946357b8aa58a1b3. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 08:00:32.156064 15775 consensus_queue.cc:1159] T 114d3eac994f466bb1787e066de05e2c P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a91b27762182454baada71e91ceab224" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20260810 08:00:32.158953 15482 ts_tablet_manager.cc:1905] T 114d3eac994f466bb1787e066de05e2c P a91b27762182454baada71e91ceab224: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 08:00:32.160830 15234 leader_election.cc:341] T d0761b090d8147b8a5537b0ed51108a9 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
W20260810 08:00:32.161453 15234 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.193:43601, user_credentials={real_user=slave}} blocked reactor thread for 89738.6us
W20260810 08:00:32.162194 15234 leader_election.cc:341] T 2d295305b5a843eaa84c8ad666355b84 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
W20260810 08:00:32.162701 15234 leader_election.cc:341] T 25de7bf64bf6450ba17b7748568bc0ed P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:32.168751 15439 tablet_bootstrap.cc:492] T 2cca8a18778c49d0be01363c31ef180b P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:32.169338 15439 ts_tablet_manager.cc:1397] T 2cca8a18778c49d0be01363c31ef180b P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.464s user 0.012s sys 0.006s
I20260810 08:00:32.172014 14681 catalog_manager.cc:4841] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): tablet 1461cdeda5664afd9c4c2b45199179b6 (table default.Table_d7227a2f586b4fd29babdc0ae25a0328 [id=5170908091794c4aa1d533a4a0f2b776]) successfully deleted
I20260810 08:00:32.172287 15783 tablet.cc:1722] T 5a22a48e2ba843b3a32a8900ab937ea0 P 3fc5e568f3724f158b616c303efb9eae: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260810 08:00:32.179605 15437 ts_tablet_manager.cc:1428] T 2cca8a18778c49d0be01363c31ef180b P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.336s user 0.021s sys 0.017s
I20260810 08:00:32.180536 15478 ts_tablet_manager.cc:1918] T 114d3eac994f466bb1787e066de05e2c P 3fc5e568f3724f158b616c303efb9eae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 08:00:32.181030 15478 log.cc:1196] T 114d3eac994f466bb1787e066de05e2c P 3fc5e568f3724f158b616c303efb9eae: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal/wals/114d3eac994f466bb1787e066de05e2c
I20260810 08:00:32.183544 15437 tablet_bootstrap.cc:492] T 25de7bf64bf6450ba17b7748568bc0ed P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:32.191903 15479 ts_tablet_manager.cc:1918] T 0fae4809ef5e4a9fb16976674e287f03 P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:32.192291 15479 log.cc:1196] T 0fae4809ef5e4a9fb16976674e287f03 P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/0fae4809ef5e4a9fb16976674e287f03
I20260810 08:00:32.192451 14988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a0e0af742340b2945be2102768b307" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:32.192934 15479 ts_tablet_manager.cc:1939] T 0fae4809ef5e4a9fb16976674e287f03 P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:32.194406 14988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4381ccb8711744ff8f9766ef49c53e28" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:32.194866 15479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43002 (request call id 69) took 3557 ms. Trace:
I20260810 08:00:32.195161 15479 rpcz_store.cc:276] 0810 08:00:28.637303 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:28.637495 (+ 192us) service_pool.cc:227] Handling call
0810 08:00:32.125839 (+3488344us) ts_tablet_manager.cc:1145] Deleting tablet 0fae4809ef5e4a9fb16976674e287f03
0810 08:00:32.125885 (+ 46us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:32.164956 (+ 39071us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:32.191622 (+ 26666us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:32.194809 (+ 3187us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3487726}
I20260810 08:00:32.196295 15479 tablet_replica.cc:330] T a1c4f8f514c94128aedd07bd5efdc3be P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:32.197033 15479 raft_consensus.cc:2229] T a1c4f8f514c94128aedd07bd5efdc3be P 9d7219686f304f62a54acc7497cbbd07 [term 1 LEADER]: Raft consensus shutting down.
W20260810 08:00:32.172914 14681 catalog_manager.cc:4821] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet 8364045e3fa2441e8dc1a1d87f73361d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8364045e3fa2441e8dc1a1d87f73361d already in progress: creating tablet
W20260810 08:00:32.206965 15079 leader_election.cc:341] T b5a0e0af742340b2945be2102768b307 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
W20260810 08:00:32.207660 15079 leader_election.cc:341] T 4381ccb8711744ff8f9766ef49c53e28 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
W20260810 08:00:32.206486 14681 catalog_manager.cc:4821] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet 3ac554bd37b045a2a8cf2743f4645f28 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ac554bd37b045a2a8cf2743f4645f28 already in progress: creating tablet
I20260810 08:00:32.208554 15479 pending_rounds.cc:62] T a1c4f8f514c94128aedd07bd5efdc3be P 9d7219686f304f62a54acc7497cbbd07: Trying to abort 1 pending ops.
I20260810 08:00:32.208797 15479 pending_rounds.cc:68] T a1c4f8f514c94128aedd07bd5efdc3be P 9d7219686f304f62a54acc7497cbbd07: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316884800533082112 op_type: NO_OP noop_request { }
I20260810 08:00:32.209158 15479 raft_consensus.cc:2866] T a1c4f8f514c94128aedd07bd5efdc3be P 9d7219686f304f62a54acc7497cbbd07 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260810 08:00:32.209471 15479 raft_consensus.cc:2258] T a1c4f8f514c94128aedd07bd5efdc3be P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:32.210065 15490 tablet_bootstrap.cc:492] T 5990e7ea243843bb869de5dde3e6cd23 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:32.210446 15490 ts_tablet_manager.cc:1397] T 5990e7ea243843bb869de5dde3e6cd23 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.343s user 0.023s sys 0.082s
I20260810 08:00:32.211205 15479 ts_tablet_manager.cc:1905] T a1c4f8f514c94128aedd07bd5efdc3be P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:32.213109 15490 raft_consensus.cc:348] T 5990e7ea243843bb869de5dde3e6cd23 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.213769 15490 raft_consensus.cc:374] T 5990e7ea243843bb869de5dde3e6cd23 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260810 08:00:32.209040 14681 catalog_manager.cc:4821] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet e162a89dcf5a4890aa5a93dfeadd131e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e162a89dcf5a4890aa5a93dfeadd131e already in progress: creating tablet
I20260810 08:00:32.214020 15490 raft_consensus.cc:729] T 5990e7ea243843bb869de5dde3e6cd23 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
W20260810 08:00:32.214753 14681 catalog_manager.cc:4808] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet a1c4f8f514c94128aedd07bd5efdc3be because the tablet was not found. No further retry: Not found: Tablet not found: a1c4f8f514c94128aedd07bd5efdc3be
I20260810 08:00:32.243078 15479 ts_tablet_manager.cc:1918] T a1c4f8f514c94128aedd07bd5efdc3be P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 08:00:32.243503 15479 log.cc:1196] T a1c4f8f514c94128aedd07bd5efdc3be P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/a1c4f8f514c94128aedd07bd5efdc3be
I20260810 08:00:32.244206 15479 ts_tablet_manager.cc:1939] T a1c4f8f514c94128aedd07bd5efdc3be P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:32.246347 15479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43002 (request call id 70) took 3515 ms. Trace:
I20260810 08:00:32.252669 15479 rpcz_store.cc:276] 0810 08:00:28.731112 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:28.731281 (+ 169us) service_pool.cc:227] Handling call
0810 08:00:32.196145 (+3464864us) ts_tablet_manager.cc:1145] Deleting tablet a1c4f8f514c94128aedd07bd5efdc3be
0810 08:00:32.196186 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:32.224490 (+ 28304us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:32.242838 (+ 18348us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:32.246281 (+ 3443us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3464282}
I20260810 08:00:32.254771 15437 tablet_bootstrap.cc:654] T 25de7bf64bf6450ba17b7748568bc0ed P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:32.255425 15479 tablet_replica.cc:330] T 924c0f73638f48a88c08d45d459132e1 P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:32.255908 15478 ts_tablet_manager.cc:1939] T 114d3eac994f466bb1787e066de05e2c P 3fc5e568f3724f158b616c303efb9eae: Deleting consensus metadata
I20260810 08:00:32.256790 15479 raft_consensus.cc:2229] T 924c0f73638f48a88c08d45d459132e1 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:32.258271 15479 raft_consensus.cc:2258] T 924c0f73638f48a88c08d45d459132e1 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:32.261281 15479 ts_tablet_manager.cc:1905] T 924c0f73638f48a88c08d45d459132e1 P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:32.264559 14681 catalog_manager.cc:4841] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): tablet 2faab63c5023431c98e7b01ce61e04c2 (table default.Table_10462271d83b4063b56a4cd0c8b6f94a [id=67e4dae7d49b48669b176960841b25ab]) successfully deleted
I20260810 08:00:32.267086 15482 ts_tablet_manager.cc:1918] T 114d3eac994f466bb1787e066de05e2c P a91b27762182454baada71e91ceab224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 08:00:32.267572 15482 log.cc:1196] T 114d3eac994f466bb1787e066de05e2c P a91b27762182454baada71e91ceab224: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal/wals/114d3eac994f466bb1787e066de05e2c
I20260810 08:00:32.268507 15482 ts_tablet_manager.cc:1939] T 114d3eac994f466bb1787e066de05e2c P a91b27762182454baada71e91ceab224: Deleting consensus metadata
W20260810 08:00:32.269413 15076 leader_election.cc:341] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:32.269894 15076 leader_election.cc:302] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:32.271603 15482 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34630 (request call id 108) took 1247 ms. Trace:
I20260810 08:00:32.276728 14960 tablet_service.cc:1514] Processing DeleteTablet for tablet d22a778bfe784b2f8a7b4d2216909f10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:26 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:32.278082 14964 tablet_service.cc:1514] Processing DeleteTablet for tablet adada21bc1bf4f19a6347d0935bd74bd with delete_type TABLET_DATA_DELETED (replaced by tablet d25928e4c39d45e69c3efeb36f03aece) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:32.279155 15482 rpcz_store.cc:276] 0810 08:00:31.023723 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:31.069330 (+ 45607us) service_pool.cc:227] Handling call
0810 08:00:31.979603 (+910273us) ts_tablet_manager.cc:1145] Deleting tablet 114d3eac994f466bb1787e066de05e2c
0810 08:00:31.979645 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:32.098341 (+118696us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b54000a1ed4. stack: 00007feaedce8033 00007feaedce7f30 00007feaed869534 00000000004db2b4 00000000004db249 00007feaf4bf4471 00007feaf4c77352 00007feaf508d26c 00007feaf5c9acfa 00007feaf5ccae10 00007feaf5cb1e0d 00007feaf5cb1db9 00007feaf5cb1d49 00007feaf5cb1d11 00007feaf5cb0f3d 0000000000580784
0810 08:00:32.176458 (+ 78117us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:32.261231 (+ 84773us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:32.271533 (+ 10302us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":267135232,"tablet-delete.queue_time_us":909403}
I20260810 08:00:32.280937 15273 tablet_service.cc:1514] Processing DeleteTablet for tablet 4381ccb8711744ff8f9766ef49c53e28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:31 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:32.283977 15111 tablet_service.cc:1514] Processing DeleteTablet for tablet 4381ccb8711744ff8f9766ef49c53e28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:31 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:32.282167 14964 tablet_service.cc:1514] Processing DeleteTablet for tablet 4381ccb8711744ff8f9766ef49c53e28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:31 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:32.288530 15111 tablet_service.cc:1514] Processing DeleteTablet for tablet adada21bc1bf4f19a6347d0935bd74bd with delete_type TABLET_DATA_DELETED (replaced by tablet d25928e4c39d45e69c3efeb36f03aece) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:32.289505 15482 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34630 (request call id 109) took 1218 ms. Trace:
I20260810 08:00:32.292697 15482 rpcz_store.cc:276] 0810 08:00:31.070745 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:31.070850 (+ 105us) service_pool.cc:227] Handling call
0810 08:00:32.289229 (+1218379us) ts_tablet_manager.cc:1145] Deleting tablet 0fae4809ef5e4a9fb16976674e287f03
0810 08:00:32.289272 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:32.289446 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1217856}
I20260810 08:00:32.294842 15479 ts_tablet_manager.cc:1918] T 924c0f73638f48a88c08d45d459132e1 P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:32.296681 14688 master_service.cc:946] ReplaceTablet: received request to replace tablet d0761b090d8147b8a5537b0ed51108a9 from {username='slave'} at 127.0.0.1:42300
I20260810 08:00:32.296916 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet d22a778bfe784b2f8a7b4d2216909f10 (table default.Table_b7ee9352799b44f4b7bbf48355b3af11 [id=f008513c7e134e1992c6c8ad3e757bbb]) successfully deleted
I20260810 08:00:32.295195 15437 tablet_bootstrap.cc:492] T 25de7bf64bf6450ba17b7748568bc0ed P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:32.298064 15437 ts_tablet_manager.cc:1397] T 25de7bf64bf6450ba17b7748568bc0ed P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.115s user 0.011s sys 0.000s
I20260810 08:00:32.254400 15439 raft_consensus.cc:348] T 2cca8a18778c49d0be01363c31ef180b P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.298594 15439 raft_consensus.cc:374] T 2cca8a18778c49d0be01363c31ef180b P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:32.299145 15439 raft_consensus.cc:729] T 2cca8a18778c49d0be01363c31ef180b P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:32.299293 15479 log.cc:1196] T 924c0f73638f48a88c08d45d459132e1 P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/924c0f73638f48a88c08d45d459132e1
I20260810 08:00:32.300019 15439 consensus_queue.cc:260] T 2cca8a18778c49d0be01363c31ef180b P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.300734 15479 ts_tablet_manager.cc:1939] T 924c0f73638f48a88c08d45d459132e1 P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:32.300405 15437 raft_consensus.cc:348] T 25de7bf64bf6450ba17b7748568bc0ed P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.301020 15437 raft_consensus.cc:374] T 25de7bf64bf6450ba17b7748568bc0ed P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:32.301250 15437 raft_consensus.cc:729] T 25de7bf64bf6450ba17b7748568bc0ed P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:32.302165 15482 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34630 (request call id 110) took 1230 ms. Trace:
I20260810 08:00:32.302542 15482 rpcz_store.cc:276] 0810 08:00:31.071771 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:31.071866 (+ 95us) service_pool.cc:227] Handling call
0810 08:00:32.301919 (+1230053us) ts_tablet_manager.cc:1145] Deleting tablet 0c152aab7e244afd857534869b13815c
0810 08:00:32.301956 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:32.302110 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1229555}
I20260810 08:00:32.302654 15479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43002 (request call id 74) took 3196 ms. Trace:
I20260810 08:00:32.304407 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet fbf62f4eab8b49f282df3ab4d95d886f (table default.Table_7f3a4e19ded64b3783a00b3d4c6839c3 [id=6c6afcf578534997878b98feb9574e2d]) successfully deleted
I20260810 08:00:32.304260 15479 rpcz_store.cc:276] 0810 08:00:29.106455 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:29.106636 (+ 181us) service_pool.cc:227] Handling call
0810 08:00:32.254912 (+3148276us) ts_tablet_manager.cc:1145] Deleting tablet 924c0f73638f48a88c08d45d459132e1
0810 08:00:32.255289 (+ 377us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:32.273870 (+ 18581us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:32.294624 (+ 20754us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:32.302597 (+ 7973us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3147542}
I20260810 08:00:32.305469 15482 tablet_replica.cc:330] T 728f8f80b3bb4b10871d8b400f25377b P a91b27762182454baada71e91ceab224: stopping tablet replica
I20260810 08:00:32.306068 15479 tablet_replica.cc:330] T 1461cdeda5664afd9c4c2b45199179b6 P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:32.306044 15267 tablet_service.cc:1514] Processing DeleteTablet for tablet adada21bc1bf4f19a6347d0935bd74bd with delete_type TABLET_DATA_DELETED (replaced by tablet d25928e4c39d45e69c3efeb36f03aece) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:32.306396 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet d0761b090d8147b8a5537b0ed51108a9 (table default.Table_f387e847280a4f048905e15c5003f58c [id=fc7978f30c9a49c8aef649a6d10d87ad]) successfully deleted
I20260810 08:00:32.303401 15437 consensus_queue.cc:260] T 25de7bf64bf6450ba17b7748568bc0ed P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.307405 15479 raft_consensus.cc:2229] T 1461cdeda5664afd9c4c2b45199179b6 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:32.307283 15723 consensus_queue.cc:1035] T 49c35b7f4a72437fa0675abe8a8f3a71 P 3fc5e568f3724f158b616c303efb9eae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 08:00:32.307837 15479 raft_consensus.cc:2258] T 1461cdeda5664afd9c4c2b45199179b6 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:32.307971 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet cb37a3a87f8b48a68c19510f306073c2 (table default.Table_64e1c230e9624fc9ba6c328346150ffe [id=46e6c2115d8c4b5e9907ed76b273638a]) successfully deleted
I20260810 08:00:32.310827 15479 ts_tablet_manager.cc:1905] T 1461cdeda5664afd9c4c2b45199179b6 P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:32.298349 15795 raft_consensus.cc:482] T 1f427f4093e5417a9ca79b1805b0675e P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 08:00:32.299312 15076 leader_election.cc:341] T 924c0f73638f48a88c08d45d459132e1 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:32.313921 15234 leader_election.cc:302] T d0761b090d8147b8a5537b0ed51108a9 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, 9d7219686f304f62a54acc7497cbbd07; no voters: a91b27762182454baada71e91ceab224
I20260810 08:00:32.313975 15076 leader_election.cc:302] T 924c0f73638f48a88c08d45d459132e1 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
W20260810 08:00:32.314716 15076 raft_consensus.cc:2670] T 924c0f73638f48a88c08d45d459132e1 P 9d7219686f304f62a54acc7497cbbd07: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260810 08:00:32.315346 15723 raft_consensus.cc:2781] T d0761b090d8147b8a5537b0ed51108a9 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:32.315730 15723 raft_consensus.cc:482] T d0761b090d8147b8a5537b0ed51108a9 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:32.316133 15723 raft_consensus.cc:3037] T d0761b090d8147b8a5537b0ed51108a9 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Advancing to term 1
W20260810 08:00:32.316797 15076 leader_election.cc:341] T 68bc39707f44457fbf57c217ad827348 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:32.317281 15076 leader_election.cc:302] T 68bc39707f44457fbf57c217ad827348 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:32.297417 15794 raft_consensus.cc:2726] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:32.318315 15794 raft_consensus.cc:2726] T 68bc39707f44457fbf57c217ad827348 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:32.318419 15482 raft_consensus.cc:2229] T 728f8f80b3bb4b10871d8b400f25377b P a91b27762182454baada71e91ceab224 [term 1 LEADER]: Raft consensus shutting down.
W20260810 08:00:32.319561 15076 leader_election.cc:341] T a244ad3bda5d45f9b89f68b734338d21 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:32.320034 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet 0fae4809ef5e4a9fb16976674e287f03 (table default.Table_beb88f8abaa447bfb6962df5718b7313 [id=41bcbdf2ef674d8b8db9284b4b0dcdb9]) successfully deleted
I20260810 08:00:32.319993 15076 leader_election.cc:302] T a244ad3bda5d45f9b89f68b734338d21 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:32.320811 15794 raft_consensus.cc:2726] T a244ad3bda5d45f9b89f68b734338d21 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:32.320921 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet a1c4f8f514c94128aedd07bd5efdc3be (table default.Table_89fe7ad723964551909b73ff6b72b9af [id=ce96e7fb7c46404a859ca08c1fd1b26a]) successfully deleted
I20260810 08:00:32.321750 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet 924c0f73638f48a88c08d45d459132e1 (table default.Table_409addd050ee45b981bddfacfab24c03 [id=76166bd7cb584067b7d2e305603a840b]) successfully deleted
I20260810 08:00:32.321926 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 110) took 1704 ms. Trace:
I20260810 08:00:32.321792 15795 raft_consensus.cc:504] T 1f427f4093e5417a9ca79b1805b0675e P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.322742 15482 raft_consensus.cc:2258] T 728f8f80b3bb4b10871d8b400f25377b P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:32.322431 15478 rpcz_store.cc:276] 0810 08:00:30.617695 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:30.747420 (+129725us) service_pool.cc:227] Handling call
0810 08:00:31.337079 (+589659us) ts_tablet_manager.cc:1145] Deleting tablet 114d3eac994f466bb1787e066de05e2c
0810 08:00:31.337122 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:31.742325 (+405203us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:32.180319 (+437994us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:32.321830 (+141511us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2901,"tablet-delete.queue_time_us":437796}
I20260810 08:00:32.324482 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 109) took 1707 ms. Trace:
I20260810 08:00:32.324594 15482 ts_tablet_manager.cc:1905] T 728f8f80b3bb4b10871d8b400f25377b P a91b27762182454baada71e91ceab224: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 08:00:32.324667 15076 leader_election.cc:341] T cb37a3a87f8b48a68c19510f306073c2 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:32.319134 15437 ts_tablet_manager.cc:1428] T 25de7bf64bf6450ba17b7748568bc0ed P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20260810 08:00:32.329779 15076 leader_election.cc:302] T cb37a3a87f8b48a68c19510f306073c2 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
W20260810 08:00:32.330476 15076 raft_consensus.cc:2670] T cb37a3a87f8b48a68c19510f306073c2 P 9d7219686f304f62a54acc7497cbbd07: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260810 08:00:32.333209 15795 leader_election.cc:288] T 1f427f4093e5417a9ca79b1805b0675e P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), a91b27762182454baada71e91ceab224 (127.14.57.193:43601)
I20260810 08:00:32.337728 14986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f427f4093e5417a9ca79b1805b0675e" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
W20260810 08:00:32.338984 15079 leader_election.cc:341] T 1f427f4093e5417a9ca79b1805b0675e P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:32.340304 15437 tablet_bootstrap.cc:492] T d0761b090d8147b8a5537b0ed51108a9 P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:32.335580 15479 ts_tablet_manager.cc:1918] T 1461cdeda5664afd9c4c2b45199179b6 P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 08:00:32.344250 15479 log.cc:1196] T 1461cdeda5664afd9c4c2b45199179b6 P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/1461cdeda5664afd9c4c2b45199179b6
I20260810 08:00:32.345103 14690 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet 3281dd757ff44d9498ef70ad372eb150
I20260810 08:00:32.345645 15479 ts_tablet_manager.cc:1939] T 1461cdeda5664afd9c4c2b45199179b6 P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:32.346062 14681 catalog_manager.cc:4841] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): tablet 114d3eac994f466bb1787e066de05e2c (table default.Table_6fcf472630534619ae18553e71ebedaf [id=2c633c1a27cf44f3ab04e66e2cba6e13]) successfully deleted
W20260810 08:00:32.346974 14681 catalog_manager.cc:4808] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet 0fae4809ef5e4a9fb16976674e287f03 because the tablet was not found. No further retry: Not found: Tablet not found: 0fae4809ef5e4a9fb16976674e287f03
I20260810 08:00:32.347587 15479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43002 (request call id 80) took 3079 ms. Trace:
W20260810 08:00:32.347775 14681 catalog_manager.cc:4808] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet 0c152aab7e244afd857534869b13815c because the tablet was not found. No further retry: Not found: Tablet not found: 0c152aab7e244afd857534869b13815c
I20260810 08:00:32.351261 15479 rpcz_store.cc:276] 0810 08:00:29.268057 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:29.335117 (+ 67060us) service_pool.cc:227] Handling call
0810 08:00:32.305930 (+2970813us) ts_tablet_manager.cc:1145] Deleting tablet 1461cdeda5664afd9c4c2b45199179b6
0810 08:00:32.305967 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:32.326642 (+ 20675us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:32.335361 (+ 8719us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:32.347530 (+ 12169us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2969909}
I20260810 08:00:32.352895 14964 tablet_service.cc:1514] Processing DeleteTablet for tablet 3281dd757ff44d9498ef70ad372eb150 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:31 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:32.354214 15437 tablet_bootstrap.cc:654] T d0761b090d8147b8a5537b0ed51108a9 P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:32.355504 15111 tablet_service.cc:1514] Processing DeleteTablet for tablet 3281dd757ff44d9498ef70ad372eb150 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:31 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:32.355631 15479 tablet_replica.cc:330] T 8364045e3fa2441e8dc1a1d87f73361d P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:32.356441 15479 raft_consensus.cc:2229] T 8364045e3fa2441e8dc1a1d87f73361d P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:32.357276 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet 1461cdeda5664afd9c4c2b45199179b6 (table default.Table_d7227a2f586b4fd29babdc0ae25a0328 [id=5170908091794c4aa1d533a4a0f2b776]) successfully deleted
I20260810 08:00:32.357343 15479 raft_consensus.cc:2258] T 8364045e3fa2441e8dc1a1d87f73361d P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:32.360091 15479 ts_tablet_manager.cc:1905] T 8364045e3fa2441e8dc1a1d87f73361d P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 08:00:32.375001 15234 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.194:36265, user_credentials={real_user=slave}} blocked reactor thread for 61276.8us
I20260810 08:00:32.374943 15795 raft_consensus.cc:482] T 561cbc6a4eb14d3584f88ea26a0fea7a P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:32.375548 15795 raft_consensus.cc:504] T 561cbc6a4eb14d3584f88ea26a0fea7a P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:32.326382 15490 consensus_queue.cc:260] T 5990e7ea243843bb869de5dde3e6cd23 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.378010 15617 raft_consensus.cc:482] T 5990e7ea243843bb869de5dde3e6cd23 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:32.378522 15617 raft_consensus.cc:504] T 5990e7ea243843bb869de5dde3e6cd23 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.379359 15490 ts_tablet_manager.cc:1428] T 5990e7ea243843bb869de5dde3e6cd23 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.169s user 0.004s sys 0.000s
I20260810 08:00:32.380378 15490 tablet_bootstrap.cc:492] T 3ac554bd37b045a2a8cf2743f4645f28 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:32.377154 15781 raft_consensus.cc:504] T d22a778bfe784b2f8a7b4d2216909f10 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.382550 15617 leader_election.cc:288] T 5990e7ea243843bb869de5dde3e6cd23 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:32.382618 15234 leader_election.cc:302] T 2d295305b5a843eaa84c8ad666355b84 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, 9d7219686f304f62a54acc7497cbbd07; no voters: a91b27762182454baada71e91ceab224
I20260810 08:00:32.383890 15795 leader_election.cc:288] T 561cbc6a4eb14d3584f88ea26a0fea7a P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:32.395936 15479 ts_tablet_manager.cc:1918] T 8364045e3fa2441e8dc1a1d87f73361d P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:32.397461 15439 ts_tablet_manager.cc:1428] T 2cca8a18778c49d0be01363c31ef180b P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.228s user 0.006s sys 0.001s
I20260810 08:00:32.398829 15479 log.cc:1196] T 8364045e3fa2441e8dc1a1d87f73361d P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/8364045e3fa2441e8dc1a1d87f73361d
I20260810 08:00:32.400019 15479 ts_tablet_manager.cc:1939] T 8364045e3fa2441e8dc1a1d87f73361d P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:32.402688 14690 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_cd979630b0c2429a848a3baa03e7e25e" } new_table_name: "default.Table_c5dc7d75cb98486986101e4dd3c9635c" modify_external_catalogs: true
I20260810 08:00:32.403844 14986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561cbc6a4eb14d3584f88ea26a0fea7a" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:32.402724 15479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43002 (request call id 83) took 2608 ms. Trace:
I20260810 08:00:32.405843 14986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5990e7ea243843bb869de5dde3e6cd23" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:32.406332 15479 rpcz_store.cc:276] 0810 08:00:29.794704 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:29.794848 (+ 144us) service_pool.cc:227] Handling call
0810 08:00:32.355473 (+2560625us) ts_tablet_manager.cc:1145] Deleting tablet 8364045e3fa2441e8dc1a1d87f73361d
0810 08:00:32.355516 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:32.380397 (+ 24881us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:32.395665 (+ 15268us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:32.402662 (+ 6997us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2559979}
I20260810 08:00:32.408252 15479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43002 (request call id 84) took 2344 ms. Trace:
I20260810 08:00:32.409575 15479 rpcz_store.cc:276] 0810 08:00:30.063753 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:30.064297 (+ 544us) service_pool.cc:227] Handling call
0810 08:00:32.407829 (+2343532us) ts_tablet_manager.cc:1145] Deleting tablet 3ac554bd37b045a2a8cf2743f4645f28
0810 08:00:32.407869 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:32.408200 (+ 331us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2342912}
I20260810 08:00:32.410732 15437 tablet_bootstrap.cc:492] T d0761b090d8147b8a5537b0ed51108a9 P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
W20260810 08:00:32.408563 15079 leader_election.cc:341] T 561cbc6a4eb14d3584f88ea26a0fea7a P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:32.412236 15437 ts_tablet_manager.cc:1397] T d0761b090d8147b8a5537b0ed51108a9 P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.072s user 0.009s sys 0.005s
W20260810 08:00:32.412587 15079 leader_election.cc:341] T 5990e7ea243843bb869de5dde3e6cd23 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:32.413249 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet 8364045e3fa2441e8dc1a1d87f73361d (table default.Table_290c3db6ca0e4fdf8e007f723b11d27d [id=97abcf4bf320496ebbec1fd2ed20765b]) successfully deleted
W20260810 08:00:32.414052 14681 catalog_manager.cc:4821] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): delete failed for tablet 3ac554bd37b045a2a8cf2743f4645f28 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ac554bd37b045a2a8cf2743f4645f28 already in progress: creating tablet
I20260810 08:00:32.414311 15479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43002 (request call id 85) took 2350 ms. Trace:
I20260810 08:00:32.416770 15479 rpcz_store.cc:276] 0810 08:00:30.064150 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:30.065023 (+ 873us) service_pool.cc:227] Handling call
0810 08:00:32.413985 (+2348962us) ts_tablet_manager.cc:1145] Deleting tablet e162a89dcf5a4890aa5a93dfeadd131e
0810 08:00:32.414025 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:32.414260 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2348433}
W20260810 08:00:32.418017 14681 catalog_manager.cc:4821] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): delete failed for tablet e162a89dcf5a4890aa5a93dfeadd131e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e162a89dcf5a4890aa5a93dfeadd131e already in progress: creating tablet
I20260810 08:00:32.418463 15479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43002 (request call id 101) took 1797 ms. Trace:
I20260810 08:00:32.418745 15479 rpcz_store.cc:276] 0810 08:00:30.621030 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:30.626447 (+ 5417us) service_pool.cc:227] Handling call
0810 08:00:32.418208 (+1791761us) ts_tablet_manager.cc:1145] Deleting tablet 0c152aab7e244afd857534869b13815c
0810 08:00:32.418247 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:32.418413 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1679337}
W20260810 08:00:32.420449 14681 catalog_manager.cc:4808] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): delete failed for tablet 0c152aab7e244afd857534869b13815c because the tablet was not found. No further retry: Not found: Tablet not found: 0c152aab7e244afd857534869b13815c
I20260810 08:00:32.421283 15479 tablet_replica.cc:330] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:32.422012 15437 raft_consensus.cc:348] T d0761b090d8147b8a5537b0ed51108a9 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.422796 15479 raft_consensus.cc:2229] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:32.423230 15479 raft_consensus.cc:2258] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:32.423584 15437 raft_consensus.cc:374] T d0761b090d8147b8a5537b0ed51108a9 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:32.424823 15479 ts_tablet_manager.cc:1905] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:32.424888 15437 raft_consensus.cc:729] T d0761b090d8147b8a5537b0ed51108a9 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:32.425731 15437 consensus_queue.cc:260] T d0761b090d8147b8a5537b0ed51108a9 P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.424589 15482 ts_tablet_manager.cc:1918] T 728f8f80b3bb4b10871d8b400f25377b P a91b27762182454baada71e91ceab224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 08:00:32.426807 15482 log.cc:1196] T 728f8f80b3bb4b10871d8b400f25377b P a91b27762182454baada71e91ceab224: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal/wals/728f8f80b3bb4b10871d8b400f25377b
I20260810 08:00:32.427678 15482 ts_tablet_manager.cc:1939] T 728f8f80b3bb4b10871d8b400f25377b P a91b27762182454baada71e91ceab224: Deleting consensus metadata
I20260810 08:00:32.432189 15795 raft_consensus.cc:482] T e0652a7684b04a89840049ce0d781199 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:32.432646 15795 raft_consensus.cc:504] T e0652a7684b04a89840049ce0d781199 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.434594 15795 leader_election.cc:288] T e0652a7684b04a89840049ce0d781199 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:32.325258 15478 rpcz_store.cc:276] 0810 08:00:30.617330 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:30.838118 (+220788us) service_pool.cc:227] Handling call
0810 08:00:32.324139 (+1486021us) ts_tablet_manager.cc:1145] Deleting tablet 2faab63c5023431c98e7b01ce61e04c2
0810 08:00:32.324200 (+ 61us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:32.324418 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1412686}
I20260810 08:00:32.442957 15482 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34630 (request call id 112) took 1369 ms. Trace:
I20260810 08:00:32.443329 15482 rpcz_store.cc:276] 0810 08:00:31.074024 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:31.077807 (+ 3783us) service_pool.cc:227] Handling call
0810 08:00:32.305315 (+1227508us) ts_tablet_manager.cc:1145] Deleting tablet 728f8f80b3bb4b10871d8b400f25377b
0810 08:00:32.305352 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:32.362469 (+ 57117us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:32.424292 (+ 61823us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:32.442885 (+ 18593us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10965,"tablet-delete.queue_time_us":1226900}
I20260810 08:00:32.444603 15482 tablet_replica.cc:330] T fbf62f4eab8b49f282df3ab4d95d886f P a91b27762182454baada71e91ceab224: stopping tablet replica
I20260810 08:00:32.445391 15482 raft_consensus.cc:2229] T fbf62f4eab8b49f282df3ab4d95d886f P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:32.436467 14986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0652a7684b04a89840049ce0d781199" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
W20260810 08:00:32.437206 14922 leader_election.cc:341] T d22a778bfe784b2f8a7b4d2216909f10 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Not found: Tablet not found: d22a778bfe784b2f8a7b4d2216909f10
I20260810 08:00:32.447324 14681 catalog_manager.cc:4841] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): tablet 728f8f80b3bb4b10871d8b400f25377b (table default.Table_4c0b6e5e875f43e088f4f0215cf39284 [id=b89eea0100b945c1851d323ce8c97591]) successfully deleted
W20260810 08:00:32.448194 15079 leader_election.cc:341] T e0652a7684b04a89840049ce0d781199 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:32.439862 15437 ts_tablet_manager.cc:1428] T d0761b090d8147b8a5537b0ed51108a9 P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.027s user 0.005s sys 0.000s
I20260810 08:00:32.434641 15781 leader_election.cc:288] T d22a778bfe784b2f8a7b4d2216909f10 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Requested vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:32.457543 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d22a778bfe784b2f8a7b4d2216909f10" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae"
I20260810 08:00:32.458228 15479 ts_tablet_manager.cc:1918] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:32.459507 15479 log.cc:1196] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/cf8bb1c44a7b4d17bddeff8b2fe81b35
I20260810 08:00:32.459892 15437 tablet_bootstrap.cc:492] T 2d295305b5a843eaa84c8ad666355b84 P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:32.460610 15479 ts_tablet_manager.cc:1939] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:32.458374 15301 raft_consensus.cc:3037] T d22a778bfe784b2f8a7b4d2216909f10 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:32.463289 15479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43002 (request call id 109) took 1316 ms. Trace:
I20260810 08:00:32.465734 15479 rpcz_store.cc:276] 0810 08:00:31.147196 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:31.148368 (+ 1172us) service_pool.cc:227] Handling call
0810 08:00:32.421130 (+1272762us) ts_tablet_manager.cc:1145] Deleting tablet cf8bb1c44a7b4d17bddeff8b2fe81b35
0810 08:00:32.421172 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:32.439923 (+ 18751us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:32.456211 (+ 16288us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:32.463231 (+ 7020us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1271853}
I20260810 08:00:32.466470 15437 tablet_bootstrap.cc:654] T 2d295305b5a843eaa84c8ad666355b84 P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:32.468003 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet cf8bb1c44a7b4d17bddeff8b2fe81b35 (table default.Table_a6ce85a21a354e7c92f158c1ea3212b0 [id=8329ae05a9b0470897f5830376c1f87a]) successfully deleted
I20260810 08:00:32.468616 15482 raft_consensus.cc:2258] T fbf62f4eab8b49f282df3ab4d95d886f P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:32.469535 15479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43002 (request call id 110) took 1215 ms. Trace:
I20260810 08:00:32.470858 15479 rpcz_store.cc:276] 0810 08:00:31.253847 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:31.254033 (+ 186us) service_pool.cc:227] Handling call
0810 08:00:32.469145 (+1215112us) ts_tablet_manager.cc:1145] Deleting tablet 20a316e404414b288a315a5386208111
0810 08:00:32.469193 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:32.469479 (+ 286us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1212146}
I20260810 08:00:32.471266 15482 ts_tablet_manager.cc:1905] T fbf62f4eab8b49f282df3ab4d95d886f P a91b27762182454baada71e91ceab224: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 08:00:32.472157 14681 catalog_manager.cc:4821] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): delete failed for tablet 20a316e404414b288a315a5386208111 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20a316e404414b288a315a5386208111 already in progress: creating tablet
I20260810 08:00:32.474280 15479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43002 (request call id 112) took 1170 ms. Trace:
I20260810 08:00:32.475687 14687 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_44d94cfdac894ff68cfbb8984912231c" } new_table_name: "default.Table_c65aa418eb904b57aab18f8b403e8c8e" modify_external_catalogs: true
I20260810 08:00:32.474583 15479 rpcz_store.cc:276] 0810 08:00:31.303393 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:31.305132 (+ 1739us) service_pool.cc:227] Handling call
0810 08:00:32.474036 (+1168904us) ts_tablet_manager.cc:1145] Deleting tablet 2cca8a18778c49d0be01363c31ef180b
0810 08:00:32.474077 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:32.474232 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1168194}
W20260810 08:00:32.477365 14681 catalog_manager.cc:4808] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): delete failed for tablet 2cca8a18778c49d0be01363c31ef180b because the tablet was not found. No further retry: Not found: Tablet not found: 2cca8a18778c49d0be01363c31ef180b
I20260810 08:00:32.478720 15479 tablet_replica.cc:330] T 4381ccb8711744ff8f9766ef49c53e28 P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:32.478307 14693 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_36a773333abe4fcbaf61c6209c63f796"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:32.479872 14693 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36a773333abe4fcbaf61c6209c63f796 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:32.480016 15479 raft_consensus.cc:2229] T 4381ccb8711744ff8f9766ef49c53e28 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:32.480432 15479 raft_consensus.cc:2258] T 4381ccb8711744ff8f9766ef49c53e28 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:32.482049 15479 ts_tablet_manager.cc:1905] T 4381ccb8711744ff8f9766ef49c53e28 P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:32.483560 14689 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_73aba277eb504cc1917f096e97f4fd50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:32.485181 14689 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73aba277eb504cc1917f096e97f4fd50 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:32.489045 14694 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_870c8e79c4574433bf1b38ff75564dc9" } modify_external_catalogs: true
I20260810 08:00:32.489745 14694 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_870c8e79c4574433bf1b38ff75564dc9" } modify_external_catalogs: true
I20260810 08:00:32.494702 14695 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_a10d38e0a2114fb4a8a59a240cd5808f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:32.495252 15437 tablet_bootstrap.cc:492] T 2d295305b5a843eaa84c8ad666355b84 P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:32.495644 15437 ts_tablet_manager.cc:1397] T 2d295305b5a843eaa84c8ad666355b84 P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.036s user 0.013s sys 0.000s
W20260810 08:00:32.496553 14695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a10d38e0a2114fb4a8a59a240cd5808f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:32.497896 15437 raft_consensus.cc:348] T 2d295305b5a843eaa84c8ad666355b84 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.498713 15437 raft_consensus.cc:374] T 2d295305b5a843eaa84c8ad666355b84 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:32.499044 15437 raft_consensus.cc:729] T 2d295305b5a843eaa84c8ad666355b84 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:32.500087 15439 tablet_bootstrap.cc:492] T 68bc39707f44457fbf57c217ad827348 P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:32.499729 15437 consensus_queue.cc:260] T 2d295305b5a843eaa84c8ad666355b84 P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.500751 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 119) took 1058 ms. Trace:
I20260810 08:00:32.501127 15478 rpcz_store.cc:276] 0810 08:00:31.442126 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:31.445118 (+ 2992us) service_pool.cc:227] Handling call
0810 08:00:32.436705 (+991587us) ts_tablet_manager.cc:1145] Deleting tablet cf8bb1c44a7b4d17bddeff8b2fe81b35
0810 08:00:32.436766 (+ 61us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:32.500690 (+ 63924us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":990834}
W20260810 08:00:32.509500 15234 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.194:36265, user_credentials={real_user=slave}} blocked reactor thread for 133871us
I20260810 08:00:32.511044 15437 ts_tablet_manager.cc:1428] T 2d295305b5a843eaa84c8ad666355b84 P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20260810 08:00:32.512708 15479 ts_tablet_manager.cc:1918] T 4381ccb8711744ff8f9766ef49c53e28 P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:32.513080 15479 log.cc:1196] T 4381ccb8711744ff8f9766ef49c53e28 P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/4381ccb8711744ff8f9766ef49c53e28
I20260810 08:00:32.513742 15479 ts_tablet_manager.cc:1939] T 4381ccb8711744ff8f9766ef49c53e28 P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:32.514765 15234 leader_election.cc:302] T 25de7bf64bf6450ba17b7748568bc0ed P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, 9d7219686f304f62a54acc7497cbbd07; no voters: a91b27762182454baada71e91ceab224
I20260810 08:00:32.520885 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet 4381ccb8711744ff8f9766ef49c53e28 (table default.Table_4f3d374da9fd4fcaaff35b3e4c3bdbf7 [id=f07affbf494748a88874b8f1edd3f7b6]) successfully deleted
I20260810 08:00:32.523092 15479 tablet_replica.cc:330] T adada21bc1bf4f19a6347d0935bd74bd P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:32.523070 15437 tablet_bootstrap.cc:492] T 561cbc6a4eb14d3584f88ea26a0fea7a P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:32.524387 15479 raft_consensus.cc:2229] T adada21bc1bf4f19a6347d0935bd74bd P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:32.526835 15479 raft_consensus.cc:2258] T adada21bc1bf4f19a6347d0935bd74bd P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:32.527976 15482 ts_tablet_manager.cc:1918] T fbf62f4eab8b49f282df3ab4d95d886f P a91b27762182454baada71e91ceab224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:32.528532 15482 log.cc:1196] T fbf62f4eab8b49f282df3ab4d95d886f P a91b27762182454baada71e91ceab224: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal/wals/fbf62f4eab8b49f282df3ab4d95d886f
I20260810 08:00:32.529047 15479 ts_tablet_manager.cc:1905] T adada21bc1bf4f19a6347d0935bd74bd P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:32.529295 15482 ts_tablet_manager.cc:1939] T fbf62f4eab8b49f282df3ab4d95d886f P a91b27762182454baada71e91ceab224: Deleting consensus metadata
I20260810 08:00:32.531488 15482 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34630 (request call id 117) took 1311 ms. Trace:
I20260810 08:00:32.531859 15482 rpcz_store.cc:276] 0810 08:00:31.219638 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:31.219747 (+ 109us) service_pool.cc:227] Handling call
0810 08:00:32.444430 (+1224683us) ts_tablet_manager.cc:1145] Deleting tablet fbf62f4eab8b49f282df3ab4d95d886f
0810 08:00:32.444478 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:32.480841 (+ 36363us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:32.527748 (+ 46907us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:32.531428 (+ 3680us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1100924}
W20260810 08:00:32.534183 14690 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a59c73c8a95f4230946357b8aa58a1b3 consensus state has no leader
I20260810 08:00:32.534617 15482 tablet_replica.cc:330] T 37d8a37a16f34a0285c43c0d71b5f072 P a91b27762182454baada71e91ceab224: stopping tablet replica
I20260810 08:00:32.535388 15482 raft_consensus.cc:2229] T 37d8a37a16f34a0285c43c0d71b5f072 P a91b27762182454baada71e91ceab224 [term 1 LEADER]: Raft consensus shutting down.
I20260810 08:00:32.536065 15482 pending_rounds.cc:62] T 37d8a37a16f34a0285c43c0d71b5f072 P a91b27762182454baada71e91ceab224: Trying to abort 1 pending ops.
I20260810 08:00:32.536338 15482 pending_rounds.cc:68] T 37d8a37a16f34a0285c43c0d71b5f072 P a91b27762182454baada71e91ceab224: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316884815906570240 op_type: NO_OP noop_request { }
I20260810 08:00:32.536769 15482 raft_consensus.cc:2866] T 37d8a37a16f34a0285c43c0d71b5f072 P a91b27762182454baada71e91ceab224 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260810 08:00:32.537127 15482 raft_consensus.cc:2258] T 37d8a37a16f34a0285c43c0d71b5f072 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:32.533802 15437 tablet_bootstrap.cc:654] T 561cbc6a4eb14d3584f88ea26a0fea7a P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:32.537709 14681 catalog_manager.cc:4841] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): tablet fbf62f4eab8b49f282df3ab4d95d886f (table default.Table_7f3a4e19ded64b3783a00b3d4c6839c3 [id=6c6afcf578534997878b98feb9574e2d]) successfully deleted
I20260810 08:00:32.541172 15482 ts_tablet_manager.cc:1905] T 37d8a37a16f34a0285c43c0d71b5f072 P a91b27762182454baada71e91ceab224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:32.553488 15437 tablet_bootstrap.cc:492] T 561cbc6a4eb14d3584f88ea26a0fea7a P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:32.553134 14690 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_5f5037d84e3f4dcc9d14d94a619e4d6a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:32.553998 15437 ts_tablet_manager.cc:1397] T 561cbc6a4eb14d3584f88ea26a0fea7a P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.008s
W20260810 08:00:32.555469 14690 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f5037d84e3f4dcc9d14d94a619e4d6a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:32.556259 15437 raft_consensus.cc:348] T 561cbc6a4eb14d3584f88ea26a0fea7a P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:32.557161 15437 raft_consensus.cc:374] T 561cbc6a4eb14d3584f88ea26a0fea7a P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:32.557437 15437 raft_consensus.cc:729] T 561cbc6a4eb14d3584f88ea26a0fea7a P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:32.559638 15437 consensus_queue.cc:260] T 561cbc6a4eb14d3584f88ea26a0fea7a P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:32.566722 15479 ts_tablet_manager.cc:1918] T adada21bc1bf4f19a6347d0935bd74bd P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:32.568182 15479 log.cc:1196] T adada21bc1bf4f19a6347d0935bd74bd P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/adada21bc1bf4f19a6347d0935bd74bd
I20260810 08:00:32.521425 15813 raft_consensus.cc:2781] T 2d295305b5a843eaa84c8ad666355b84 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Leader election won for term 1
I20260810 08:00:32.536060 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 120) took 1089 ms. Trace:
I20260810 08:00:32.569571 15479 ts_tablet_manager.cc:1939] T adada21bc1bf4f19a6347d0935bd74bd P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:32.569171 15478 rpcz_store.cc:276] 0810 08:00:31.448079 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:31.449198 (+ 1119us) service_pool.cc:227] Handling call
0810 08:00:32.502047 (+1052849us) ts_tablet_manager.cc:1145] Deleting tablet 20a316e404414b288a315a5386208111
0810 08:00:32.502088 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:32.535995 (+ 33907us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1052177}
I20260810 08:00:32.536672 15807 raft_consensus.cc:482] T 2cca8a18778c49d0be01363c31ef180b P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:32.571813 15437 ts_tablet_manager.cc:1428] T 561cbc6a4eb14d3584f88ea26a0fea7a P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20260810 08:00:32.572845 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet adada21bc1bf4f19a6347d0935bd74bd (table default.Table_93c814deac714a3288f3980340acbc65 [id=4c70468b9e8649849cd0e79d59be6bb9]) successfully deleted
W20260810 08:00:32.575309 14681 catalog_manager.cc:4821] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): delete failed for tablet 3281dd757ff44d9498ef70ad372eb150 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3281dd757ff44d9498ef70ad372eb150 already in progress: creating tablet
I20260810 08:00:32.539942 15723 raft_consensus.cc:504] T d0761b090d8147b8a5537b0ed51108a9 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.586829 15437 tablet_bootstrap.cc:492] T 4381ccb8711744ff8f9766ef49c53e28 P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:32.593139 15482 ts_tablet_manager.cc:1918] T 37d8a37a16f34a0285c43c0d71b5f072 P a91b27762182454baada71e91ceab224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 08:00:32.594666 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 121) took 1144 ms. Trace:
I20260810 08:00:32.595449 15478 rpcz_store.cc:276] 0810 08:00:31.450436 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:31.456391 (+ 5955us) service_pool.cc:227] Handling call
0810 08:00:32.570273 (+1113882us) ts_tablet_manager.cc:1145] Deleting tablet 35b6da60a5544d7e9b657a5f1e6f90ff
0810 08:00:32.570332 (+ 59us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:32.594524 (+ 24192us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1112277}
I20260810 08:00:32.599910 15482 log.cc:1196] T 37d8a37a16f34a0285c43c0d71b5f072 P a91b27762182454baada71e91ceab224: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal/wals/37d8a37a16f34a0285c43c0d71b5f072
I20260810 08:00:32.600800 15482 ts_tablet_manager.cc:1939] T 37d8a37a16f34a0285c43c0d71b5f072 P a91b27762182454baada71e91ceab224: Deleting consensus metadata
I20260810 08:00:32.601652 15482 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34630 (request call id 118) took 1381 ms. Trace:
I20260810 08:00:32.602142 15482 rpcz_store.cc:276] 0810 08:00:31.220421 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:31.220531 (+ 110us) service_pool.cc:227] Handling call
0810 08:00:32.534480 (+1313949us) ts_tablet_manager.cc:1145] Deleting tablet 37d8a37a16f34a0285c43c0d71b5f072
0810 08:00:32.534519 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:32.561237 (+ 26718us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:32.592849 (+ 31612us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:32.601567 (+ 8718us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1181398}
I20260810 08:00:32.599601 15437 tablet_bootstrap.cc:654] T 4381ccb8711744ff8f9766ef49c53e28 P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:32.605877 14681 catalog_manager.cc:4841] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): tablet 37d8a37a16f34a0285c43c0d71b5f072 (table default.Table_bae5256f457944049d86e384fb3759d6 [id=dad0c6e295df4fdd9b2c27d2b19fbc5f]) successfully deleted
I20260810 08:00:32.606990 15482 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34630 (request call id 122) took 1298 ms. Trace:
I20260810 08:00:32.607316 15482 rpcz_store.cc:276] 0810 08:00:31.308415 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:31.308595 (+ 180us) service_pool.cc:227] Handling call
0810 08:00:32.606652 (+1298057us) ts_tablet_manager.cc:1145] Deleting tablet cf8bb1c44a7b4d17bddeff8b2fe81b35
0810 08:00:32.606690 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:32.606942 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1229201}
W20260810 08:00:32.608490 14681 catalog_manager.cc:4821] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet cf8bb1c44a7b4d17bddeff8b2fe81b35 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf8bb1c44a7b4d17bddeff8b2fe81b35 already in progress: creating tablet
I20260810 08:00:32.609508 15482 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34630 (request call id 123) took 1300 ms. Trace:
I20260810 08:00:32.610196 15482 rpcz_store.cc:276] 0810 08:00:31.309465 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:31.309641 (+ 176us) service_pool.cc:227] Handling call
0810 08:00:32.609203 (+1299562us) ts_tablet_manager.cc:1145] Deleting tablet 20a316e404414b288a315a5386208111
0810 08:00:32.609238 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:32.609462 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1231424}
W20260810 08:00:32.611964 14681 catalog_manager.cc:4821] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet 20a316e404414b288a315a5386208111 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20a316e404414b288a315a5386208111 already in progress: creating tablet
I20260810 08:00:32.612933 15482 tablet_replica.cc:330] T d22a778bfe784b2f8a7b4d2216909f10 P a91b27762182454baada71e91ceab224: stopping tablet replica
I20260810 08:00:32.613360 15490 tablet_bootstrap.cc:654] T 3ac554bd37b045a2a8cf2743f4645f28 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:32.614195 15482 raft_consensus.cc:2229] T d22a778bfe784b2f8a7b4d2216909f10 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:32.615423 15482 raft_consensus.cc:2258] T d22a778bfe784b2f8a7b4d2216909f10 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:32.617439 15482 ts_tablet_manager.cc:1905] T d22a778bfe784b2f8a7b4d2216909f10 P a91b27762182454baada71e91ceab224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:32.636497 15437 tablet_bootstrap.cc:492] T 4381ccb8711744ff8f9766ef49c53e28 P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:32.637025 15437 ts_tablet_manager.cc:1397] T 4381ccb8711744ff8f9766ef49c53e28 P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.050s user 0.005s sys 0.009s
I20260810 08:00:32.639448 15437 raft_consensus.cc:348] T 4381ccb8711744ff8f9766ef49c53e28 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.640327 15437 raft_consensus.cc:374] T 4381ccb8711744ff8f9766ef49c53e28 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:32.640667 15437 raft_consensus.cc:729] T 4381ccb8711744ff8f9766ef49c53e28 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:32.642248 15437 consensus_queue.cc:260] T 4381ccb8711744ff8f9766ef49c53e28 P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.654343 15437 ts_tablet_manager.cc:1428] T 4381ccb8711744ff8f9766ef49c53e28 P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20260810 08:00:32.659588 15437 tablet_bootstrap.cc:492] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:32.670482 15437 tablet_bootstrap.cc:654] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:32.592123 15807 raft_consensus.cc:504] T 2cca8a18778c49d0be01363c31ef180b P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.603446 15815 raft_consensus.cc:2781] T 25de7bf64bf6450ba17b7748568bc0ed P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Leader election won for term 1
I20260810 08:00:32.672461 15815 raft_consensus.cc:686] T 25de7bf64bf6450ba17b7748568bc0ed P 3fc5e568f3724f158b616c303efb9eae [term 1 LEADER]: Becoming Leader. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Running, Role: LEADER
I20260810 08:00:32.603842 15813 raft_consensus.cc:686] T 2d295305b5a843eaa84c8ad666355b84 P 3fc5e568f3724f158b616c303efb9eae [term 1 LEADER]: Becoming Leader. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Running, Role: LEADER
I20260810 08:00:32.673731 15807 leader_election.cc:288] T 2cca8a18778c49d0be01363c31ef180b P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:32.674182 15813 consensus_queue.cc:237] T 2d295305b5a843eaa84c8ad666355b84 P 3fc5e568f3724f158b616c303efb9eae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.674966 15478 tablet_replica.cc:330] T 37d8a37a16f34a0285c43c0d71b5f072 P 3fc5e568f3724f158b616c303efb9eae: stopping tablet replica
I20260810 08:00:32.675915 15478 raft_consensus.cc:2229] T 37d8a37a16f34a0285c43c0d71b5f072 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:32.676421 15478 raft_consensus.cc:2258] T 37d8a37a16f34a0285c43c0d71b5f072 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:32.673292 15815 consensus_queue.cc:237] T 25de7bf64bf6450ba17b7748568bc0ed P 3fc5e568f3724f158b616c303efb9eae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.679527 15482 ts_tablet_manager.cc:1918] T d22a778bfe784b2f8a7b4d2216909f10 P a91b27762182454baada71e91ceab224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:32.679876 15482 log.cc:1196] T d22a778bfe784b2f8a7b4d2216909f10 P a91b27762182454baada71e91ceab224: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal/wals/d22a778bfe784b2f8a7b4d2216909f10
I20260810 08:00:32.680509 15482 ts_tablet_manager.cc:1939] T d22a778bfe784b2f8a7b4d2216909f10 P a91b27762182454baada71e91ceab224: Deleting consensus metadata
I20260810 08:00:32.604128 15723 leader_election.cc:288] T d0761b090d8147b8a5537b0ed51108a9 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Requested vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:32.622036 15783 tablet.cc:1717] T 5a22a48e2ba843b3a32a8900ab937ea0 P 3fc5e568f3724f158b616c303efb9eae: Skipping requested alter to schema version 3, tablet already version 3
I20260810 08:00:32.634160 15301 raft_consensus.cc:2454] T d22a778bfe784b2f8a7b4d2216909f10 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 1.
I20260810 08:00:32.685014 14918 leader_election.cc:302] T d22a778bfe784b2f8a7b4d2216909f10 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224; no voters: 9d7219686f304f62a54acc7497cbbd07
W20260810 08:00:32.685653 14918 raft_consensus.cc:2670] T d22a778bfe784b2f8a7b4d2216909f10 P a91b27762182454baada71e91ceab224: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260810 08:00:32.691828 15437 tablet_bootstrap.cc:492] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:32.692317 15437 ts_tablet_manager.cc:1397] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.033s user 0.008s sys 0.000s
W20260810 08:00:32.693982 14687 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0687d323788642f798465ece973702bb consensus state has no leader
I20260810 08:00:32.694728 14681 catalog_manager.cc:4841] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): tablet d22a778bfe784b2f8a7b4d2216909f10 (table default.Table_b7ee9352799b44f4b7bbf48355b3af11 [id=f008513c7e134e1992c6c8ad3e757bbb]) successfully deleted
I20260810 08:00:32.695022 14687 master_service.cc:946] ReplaceTablet: received request to replace tablet 68048e2768cf441d906e675254175c9c from {username='slave'} at 127.0.0.1:42300
W20260810 08:00:32.696696 14681 catalog_manager.cc:4821] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet adada21bc1bf4f19a6347d0935bd74bd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet adada21bc1bf4f19a6347d0935bd74bd already in progress: creating tablet
I20260810 08:00:32.698208 15437 raft_consensus.cc:348] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.699065 15437 raft_consensus.cc:374] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:32.699368 15437 raft_consensus.cc:729] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:32.700388 15482 tablet_replica.cc:330] T 4381ccb8711744ff8f9766ef49c53e28 P a91b27762182454baada71e91ceab224: stopping tablet replica
I20260810 08:00:32.701187 15482 raft_consensus.cc:2229] T 4381ccb8711744ff8f9766ef49c53e28 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:32.701594 15482 raft_consensus.cc:2258] T 4381ccb8711744ff8f9766ef49c53e28 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:32.702337 15439 tablet_bootstrap.cc:654] T 68bc39707f44457fbf57c217ad827348 P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
W20260810 08:00:32.709414 15234 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.194:36265, user_credentials={real_user=slave}} blocked reactor thread for 194737us
I20260810 08:00:32.711161 15302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4381ccb8711744ff8f9766ef49c53e28" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:32.713234 15302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f427f4093e5417a9ca79b1805b0675e" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:32.715184 15302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5990e7ea243843bb869de5dde3e6cd23" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:32.716980 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a0e0af742340b2945be2102768b307" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:32.706319 15437 consensus_queue.cc:260] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.749507 15490 tablet_bootstrap.cc:492] T 3ac554bd37b045a2a8cf2743f4645f28 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:32.749975 15490 ts_tablet_manager.cc:1397] T 3ac554bd37b045a2a8cf2743f4645f28 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.370s user 0.002s sys 0.009s
I20260810 08:00:32.709903 15781 raft_consensus.cc:482] T 2cca8a18778c49d0be01363c31ef180b P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:32.751300 15781 raft_consensus.cc:504] T 2cca8a18778c49d0be01363c31ef180b P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.752337 15490 raft_consensus.cc:348] T 3ac554bd37b045a2a8cf2743f4645f28 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.752977 15490 raft_consensus.cc:374] T 3ac554bd37b045a2a8cf2743f4645f28 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:32.753222 15490 raft_consensus.cc:729] T 3ac554bd37b045a2a8cf2743f4645f28 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:32.755129 15482 ts_tablet_manager.cc:1905] T 4381ccb8711744ff8f9766ef49c53e28 P a91b27762182454baada71e91ceab224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:32.758553 15490 consensus_queue.cc:260] T 3ac554bd37b045a2a8cf2743f4645f28 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.769230 15490 ts_tablet_manager.cc:1428] T 3ac554bd37b045a2a8cf2743f4645f28 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20260810 08:00:32.771497 15490 tablet_bootstrap.cc:492] T 579bac622ddf40bf80d01d66ae2fa100 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:32.776659 15490 tablet_bootstrap.cc:654] T 579bac622ddf40bf80d01d66ae2fa100 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:32.788599 15781 leader_election.cc:288] T 2cca8a18778c49d0be01363c31ef180b P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
W20260810 08:00:32.791993 14922 leader_election.cc:341] T 2cca8a18778c49d0be01363c31ef180b P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Not found: Tablet not found: 2cca8a18778c49d0be01363c31ef180b
I20260810 08:00:32.797466 14678 catalog_manager.cc:4841] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): tablet 114d3eac994f466bb1787e066de05e2c (table default.Table_6fcf472630534619ae18553e71ebedaf [id=2c633c1a27cf44f3ab04e66e2cba6e13]) successfully deleted
W20260810 08:00:32.798448 14678 catalog_manager.cc:4808] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet 2faab63c5023431c98e7b01ce61e04c2 because the tablet was not found. No further retry: Not found: Tablet not found: 2faab63c5023431c98e7b01ce61e04c2
W20260810 08:00:32.799118 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet cf8bb1c44a7b4d17bddeff8b2fe81b35 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf8bb1c44a7b4d17bddeff8b2fe81b35 already in progress: creating tablet
I20260810 08:00:32.798924 14986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0761b090d8147b8a5537b0ed51108a9" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224"
I20260810 08:00:32.799712 14986 raft_consensus.cc:3037] T d0761b090d8147b8a5537b0ed51108a9 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
W20260810 08:00:32.799729 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet 20a316e404414b288a315a5386208111 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20a316e404414b288a315a5386208111 already in progress: creating tablet
I20260810 08:00:32.800400 15490 tablet_bootstrap.cc:492] T 579bac622ddf40bf80d01d66ae2fa100 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:32.800804 15490 ts_tablet_manager.cc:1397] T 579bac622ddf40bf80d01d66ae2fa100 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.007s
I20260810 08:00:32.801252 15478 ts_tablet_manager.cc:1905] T 37d8a37a16f34a0285c43c0d71b5f072 P 3fc5e568f3724f158b616c303efb9eae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:32.803542 14988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cca8a18778c49d0be01363c31ef180b" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:32.804141 14988 raft_consensus.cc:2454] T 2cca8a18778c49d0be01363c31ef180b P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 0.
I20260810 08:00:32.803056 15490 raft_consensus.cc:348] T 579bac622ddf40bf80d01d66ae2fa100 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.808012 15490 raft_consensus.cc:374] T 579bac622ddf40bf80d01d66ae2fa100 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:32.808279 15490 raft_consensus.cc:729] T 579bac622ddf40bf80d01d66ae2fa100 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:32.808637 15302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0652a7684b04a89840049ce0d781199" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:32.808857 15490 consensus_queue.cc:260] T 579bac622ddf40bf80d01d66ae2fa100 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.810719 15302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cca8a18778c49d0be01363c31ef180b" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:32.811331 15302 raft_consensus.cc:2454] T 2cca8a18778c49d0be01363c31ef180b P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 0.
I20260810 08:00:32.812669 14918 leader_election.cc:302] T 2cca8a18778c49d0be01363c31ef180b P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224; no voters: 9d7219686f304f62a54acc7497cbbd07
I20260810 08:00:32.814522 15834 raft_consensus.cc:2781] T 2cca8a18778c49d0be01363c31ef180b P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:32.815055 15490 ts_tablet_manager.cc:1428] T 579bac622ddf40bf80d01d66ae2fa100 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20260810 08:00:32.815804 15834 raft_consensus.cc:482] T 2cca8a18778c49d0be01363c31ef180b P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:32.816226 15834 raft_consensus.cc:3037] T 2cca8a18778c49d0be01363c31ef180b P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:32.815793 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561cbc6a4eb14d3584f88ea26a0fea7a" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:32.817184 15490 tablet_bootstrap.cc:492] T 8e603d04f7454501b2edfb5a1f4ea7f0 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:32.818763 14693 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_3d528c5b4aa54991bf128ada67924546" } modify_external_catalogs: true
I20260810 08:00:32.819303 14693 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_3d528c5b4aa54991bf128ada67924546" } modify_external_catalogs: true
I20260810 08:00:32.815560 14986 raft_consensus.cc:2454] T d0761b090d8147b8a5537b0ed51108a9 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 1.
I20260810 08:00:32.824370 14689 master_service.cc:946] ReplaceTablet: received request to replace tablet cb37a3a87f8b48a68c19510f306073c2 from {username='slave'} at 127.0.0.1:42300
I20260810 08:00:32.826236 14689 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_28276fa7c9a042209ceaf8cc7c13eb43" } new_table_name: "default.Table_4f3c632faa1e4c2d86acf10a656ee7c7" modify_external_catalogs: true
I20260810 08:00:32.828871 15437 ts_tablet_manager.cc:1428] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.136s user 0.010s sys 0.000s
I20260810 08:00:32.832679 15437 tablet_bootstrap.cc:492] T cb37a3a87f8b48a68c19510f306073c2 P a91b27762182454baada71e91ceab224: Bootstrap starting.
W20260810 08:00:32.834357 15076 leader_election.cc:341] T 4381ccb8711744ff8f9766ef49c53e28 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:32.834806 15076 leader_election.cc:302] T 4381ccb8711744ff8f9766ef49c53e28 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
W20260810 08:00:32.835409 15076 raft_consensus.cc:2670] T 4381ccb8711744ff8f9766ef49c53e28 P 9d7219686f304f62a54acc7497cbbd07: unable to run election callback: Service unavailable: Thread pool token was shut down
W20260810 08:00:32.836226 14678 catalog_manager.cc:4808] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet 35b6da60a5544d7e9b657a5f1e6f90ff because the tablet was not found. No further retry: Not found: Tablet not found: 35b6da60a5544d7e9b657a5f1e6f90ff
W20260810 08:00:32.838088 15076 leader_election.cc:341] T 1f427f4093e5417a9ca79b1805b0675e P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:32.838594 15076 leader_election.cc:302] T 1f427f4093e5417a9ca79b1805b0675e P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:32.839453 15795 raft_consensus.cc:2726] T 1f427f4093e5417a9ca79b1805b0675e P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 08:00:32.842454 15076 leader_election.cc:341] T 5990e7ea243843bb869de5dde3e6cd23 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:32.842945 15076 leader_election.cc:302] T 5990e7ea243843bb869de5dde3e6cd23 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:32.843736 15795 raft_consensus.cc:2726] T 5990e7ea243843bb869de5dde3e6cd23 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 08:00:32.845312 15076 leader_election.cc:341] T b5a0e0af742340b2945be2102768b307 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:32.845467 15834 raft_consensus.cc:504] T 2cca8a18778c49d0be01363c31ef180b P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.845777 15076 leader_election.cc:302] T b5a0e0af742340b2945be2102768b307 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:32.852279 15795 raft_consensus.cc:2726] T b5a0e0af742340b2945be2102768b307 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:32.856817 15437 tablet_bootstrap.cc:654] T cb37a3a87f8b48a68c19510f306073c2 P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:32.843232 15490 tablet_bootstrap.cc:654] T 8e603d04f7454501b2edfb5a1f4ea7f0 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
W20260810 08:00:32.862406 14679 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 0687d323788642f798465ece973702bb (table default.Table_c65aa418eb904b57aab18f8b403e8c8e [id=0d60b763c4804818ae26fa693a4fd808], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 0687d323788642f798465ece973702bb consensus state has no leader
I20260810 08:00:32.858968 15439 tablet_bootstrap.cc:492] T 68bc39707f44457fbf57c217ad827348 P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:32.863662 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cca8a18778c49d0be01363c31ef180b" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae"
I20260810 08:00:32.864321 15301 raft_consensus.cc:3037] T 2cca8a18778c49d0be01363c31ef180b P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:32.870061 15439 ts_tablet_manager.cc:1397] T 68bc39707f44457fbf57c217ad827348 P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.414s user 0.009s sys 0.004s
W20260810 08:00:32.870476 14922 leader_election.cc:341] T 2cca8a18778c49d0be01363c31ef180b P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Not found: Tablet not found: 2cca8a18778c49d0be01363c31ef180b
I20260810 08:00:32.871068 15834 leader_election.cc:288] T 2cca8a18778c49d0be01363c31ef180b P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Requested vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:32.875125 15439 raft_consensus.cc:348] T 68bc39707f44457fbf57c217ad827348 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.875885 15439 raft_consensus.cc:374] T 68bc39707f44457fbf57c217ad827348 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260810 08:00:32.879660 14681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.57.252:33819: connect: Connection refused (error 111) [suppressed 7 similar messages]
I20260810 08:00:32.883896 15437 tablet_bootstrap.cc:492] T cb37a3a87f8b48a68c19510f306073c2 P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:32.884330 15437 ts_tablet_manager.cc:1397] T cb37a3a87f8b48a68c19510f306073c2 P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.052s user 0.005s sys 0.004s
I20260810 08:00:32.886540 15437 raft_consensus.cc:348] T cb37a3a87f8b48a68c19510f306073c2 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.887261 15437 raft_consensus.cc:374] T cb37a3a87f8b48a68c19510f306073c2 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:32.887506 15437 raft_consensus.cc:729] T cb37a3a87f8b48a68c19510f306073c2 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:32.887809 15490 tablet_bootstrap.cc:492] T 8e603d04f7454501b2edfb5a1f4ea7f0 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:32.888322 15490 ts_tablet_manager.cc:1397] T 8e603d04f7454501b2edfb5a1f4ea7f0 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.071s user 0.008s sys 0.000s
I20260810 08:00:32.889711 15482 ts_tablet_manager.cc:1918] T 4381ccb8711744ff8f9766ef49c53e28 P a91b27762182454baada71e91ceab224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:32.890050 15482 log.cc:1196] T 4381ccb8711744ff8f9766ef49c53e28 P a91b27762182454baada71e91ceab224: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal/wals/4381ccb8711744ff8f9766ef49c53e28
I20260810 08:00:32.890654 15482 ts_tablet_manager.cc:1939] T 4381ccb8711744ff8f9766ef49c53e28 P a91b27762182454baada71e91ceab224: Deleting consensus metadata
I20260810 08:00:32.892930 15437 consensus_queue.cc:260] T cb37a3a87f8b48a68c19510f306073c2 P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.899369 15437 ts_tablet_manager.cc:1428] T cb37a3a87f8b48a68c19510f306073c2 P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.015s user 0.003s sys 0.004s
I20260810 08:00:32.899497 15490 raft_consensus.cc:348] T 8e603d04f7454501b2edfb5a1f4ea7f0 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.900187 15490 raft_consensus.cc:374] T 8e603d04f7454501b2edfb5a1f4ea7f0 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:32.900450 15490 raft_consensus.cc:729] T 8e603d04f7454501b2edfb5a1f4ea7f0 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:32.902971 14681 catalog_manager.cc:4841] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): tablet 4381ccb8711744ff8f9766ef49c53e28 (table default.Table_4f3d374da9fd4fcaaff35b3e4c3bdbf7 [id=f07affbf494748a88874b8f1edd3f7b6]) successfully deleted
I20260810 08:00:32.904453 15437 tablet_bootstrap.cc:492] T 924c0f73638f48a88c08d45d459132e1 P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:32.905284 15439 raft_consensus.cc:729] T 68bc39707f44457fbf57c217ad827348 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:32.895427 15301 raft_consensus.cc:2454] T 2cca8a18778c49d0be01363c31ef180b P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 1.
W20260810 08:00:32.860648 15234 consensus_peers.cc:487] T fbf62f4eab8b49f282df3ab4d95d886f P 3fc5e568f3724f158b616c303efb9eae -> Peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Couldn't send request to peer a91b27762182454baada71e91ceab224. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fbf62f4eab8b49f282df3ab4d95d886f. This is attempt 1: this message will repeat every 5th retry.
W20260810 08:00:32.906672 15234 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.57.193:43601, user_credentials={real_user=slave}} blocked reactor thread for 46126.3us
I20260810 08:00:32.907172 14918 leader_election.cc:302] T 2cca8a18778c49d0be01363c31ef180b P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224; no voters: 9d7219686f304f62a54acc7497cbbd07
I20260810 08:00:32.907486 15234 leader_election.cc:302] T 2cca8a18778c49d0be01363c31ef180b P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224; no voters:
I20260810 08:00:32.908223 15234 leader_election.cc:302] T d0761b090d8147b8a5537b0ed51108a9 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224; no voters:
I20260810 08:00:32.908315 15834 raft_consensus.cc:2781] T 2cca8a18778c49d0be01363c31ef180b P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 08:00:32.908730 15834 raft_consensus.cc:686] T 2cca8a18778c49d0be01363c31ef180b P a91b27762182454baada71e91ceab224 [term 1 LEADER]: Becoming Leader. State: Replica: a91b27762182454baada71e91ceab224, State: Running, Role: LEADER
I20260810 08:00:32.909036 15818 raft_consensus.cc:2741] T 2cca8a18778c49d0be01363c31ef180b P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260810 08:00:32.909600 15818 raft_consensus.cc:2781] T d0761b090d8147b8a5537b0ed51108a9 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Leader election won for term 1
I20260810 08:00:32.910063 15818 raft_consensus.cc:686] T d0761b090d8147b8a5537b0ed51108a9 P 3fc5e568f3724f158b616c303efb9eae [term 1 LEADER]: Becoming Leader. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Running, Role: LEADER
I20260810 08:00:32.910050 15834 consensus_queue.cc:237] T 2cca8a18778c49d0be01363c31ef180b P a91b27762182454baada71e91ceab224 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.910815 15818 consensus_queue.cc:237] T d0761b090d8147b8a5537b0ed51108a9 P 3fc5e568f3724f158b616c303efb9eae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.917431 15275 tablet_service.cc:1514] Processing DeleteTablet for tablet 3281dd757ff44d9498ef70ad372eb150 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:31 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:32.917946 15490 consensus_queue.cc:260] T 8e603d04f7454501b2edfb5a1f4ea7f0 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
W20260810 08:00:32.918385 15076 leader_election.cc:341] T e0652a7684b04a89840049ce0d781199 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:32.919044 15076 leader_election.cc:302] T e0652a7684b04a89840049ce0d781199 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:32.921797 15795 raft_consensus.cc:2726] T e0652a7684b04a89840049ce0d781199 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:32.922724 14688 catalog_manager.cc:5587] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729) does not have the latest schema for tablet 49c35b7f4a72437fa0675abe8a8f3a71 (table default.Table_1567aaa1d2ab44028745e82c29232682 [id=6cf63d3e931a4f6487074494254805bb]). Expected version 4 got 0
I20260810 08:00:32.923216 15439 consensus_queue.cc:260] T 68bc39707f44457fbf57c217ad827348 P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.928347 15831 raft_consensus.cc:482] T 2d295305b5a843eaa84c8ad666355b84 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:32.928833 15831 raft_consensus.cc:504] T 2d295305b5a843eaa84c8ad666355b84 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:32.930966 15831 leader_election.cc:288] T 2d295305b5a843eaa84c8ad666355b84 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:32.931429 15490 ts_tablet_manager.cc:1428] T 8e603d04f7454501b2edfb5a1f4ea7f0 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.043s user 0.007s sys 0.001s
I20260810 08:00:32.932793 15439 ts_tablet_manager.cc:1428] T 68bc39707f44457fbf57c217ad827348 P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.060s user 0.005s sys 0.000s
I20260810 08:00:32.933068 15437 tablet_bootstrap.cc:654] T 924c0f73638f48a88c08d45d459132e1 P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:32.934641 15490 tablet_bootstrap.cc:492] T 8a8454f693494ab3b82f5b85c2d62bb2 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:32.937068 14692 catalog_manager.cc:5587] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265) does not have the latest schema for tablet b5a0e0af742340b2945be2102768b307 (table default.Table_eab0fdadaf49486ca724a0a6e179a889 [id=fc52022e20044cc3bc88ac143e088e69]). Expected version 2 got 0
I20260810 08:00:32.937124 15144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d295305b5a843eaa84c8ad666355b84" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:32.937968 14686 catalog_manager.cc:6040] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: default.Table_8aecb3c6298849c38bcafee965161494 [id=ac3a5787385940bca8030d7bac893a27] alter complete (version 3)
I20260810 08:00:32.938197 15144 raft_consensus.cc:2379] T 2d295305b5a843eaa84c8ad666355b84 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a91b27762182454baada71e91ceab224 in current term 1: Already voted for candidate 3fc5e568f3724f158b616c303efb9eae in this term.
I20260810 08:00:32.938779 15856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.57.252:33819 every 8 connection(s)
W20260810 08:00:32.941238 14686 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.14.57.193:35941 (request call id 36) took 830 ms (client timeout 1000 ms). Trace:
I20260810 08:00:32.943504 15454 rpc_server.cc:307] RPC server started. Bound to: 127.14.57.252:33819
I20260810 08:00:32.943725 14964 tablet_service.cc:1467] Processing CreateTablet for tablet 521d16ada2424d3aa41f838a511131fb (DEFAULT_TABLE table=default.Table_6cbd44cdd6a44bc4b3eb991d874353eb [id=5e6a4db5ec064eac935634a179729b1f]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 08:00:32.941598 14686 rpcz_store.cc:269] 0810 08:00:32.110481 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:32.173187 (+ 62706us) service_pool.cc:227] Handling call
0810 08:00:32.230997 (+ 57810us) spinlock_profiling.cc:243] Waited 55.7 ms on lock 0x7b5400021554. stack: 00007fe2372c4033 00007fe2372c3f30 00007fe236e45534 00000000004db2b4 00000000004db249 00007fe23e1d0af0 00007fe23e253f2a 00007fe24011fd6c 00007fe240219c3e 00007fe23dd50fb4 00007fe23dd50f42 00007fe23dd50e71 00007fe23dd50dec 00007fe23dd500a2 00007fe23bffa94c 00007fe23bff9d86
0810 08:00:32.941164 (+710167us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 08:00:32.333090 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 08:00:32.333449 (+ 359us) write_op.cc:432] Acquiring schema lock in shared mode
0810 08:00:32.333479 (+ 30us) write_op.cc:435] Acquired schema lock
0810 08:00:32.333496 (+ 17us) tablet.cc:660] Decoding operations
0810 08:00:32.333790 (+ 294us) write_op.cc:620] Acquiring the partition lock for write op
0810 08:00:32.333826 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0810 08:00:32.333848 (+ 22us) tablet.cc:683] Acquiring locks for 1 operations
0810 08:00:32.383771 (+ 49923us) tablet.cc:699] Row locks acquired
0810 08:00:32.383789 (+ 18us) write_op.cc:260] PREPARE: finished
0810 08:00:32.383934 (+ 145us) write_op.cc:270] Start()
0810 08:00:32.384022 (+ 88us) write_op.cc:276] Timestamp: P: 1786348832383733 usec, L: 0
0810 08:00:32.384038 (+ 16us) op_driver.cc:331] REPLICATION: starting
0810 08:00:32.400661 (+ 16623us) log.cc:842] Serialized 356 byte log entry
0810 08:00:32.407103 (+ 6442us) spinlock_profiling.cc:243] Waited 16 ms on lock 0x7b5400021554. stack: 00007fe2372c4033 00007fe2372c3f30 00007fe236e45534 00000000004db2b4 00000000004db249 00007fe23e1d0471 00007fe23e247198 00007fe23e6a8ade 00007fe23e6a82b8 00007fe23e6acba1 00007fe23e6acb59 00007fe23e6acae9 00007fe23e6acab1 00007fe23e6abdad 0000000000580784 00000000005805b9
0810 08:00:32.418944 (+ 11841us) op_driver.cc:444] REPLICATION: finished
0810 08:00:32.419509 (+ 565us) write_op.cc:301] APPLY: starting
0810 08:00:32.419564 (+ 55us) tablet.cc:1365] starting BulkCheckPresence
0810 08:00:32.419680 (+ 116us) tablet.cc:1368] finished BulkCheckPresence
0810 08:00:32.419711 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 08:00:32.419930 (+ 219us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 08:00:32.420127 (+ 197us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 08:00:32.420190 (+ 63us) write_op.cc:312] APPLY: finished
0810 08:00:32.420401 (+ 211us) log.cc:842] Serialized 23 byte log entry
0810 08:00:32.420596 (+ 195us) write_op.cc:489] Releasing partition, row and schema locks
0810 08:00:32.420681 (+ 85us) write_op.cc:454] Released schema lock
0810 08:00:32.420722 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0810 08:00:32.420958 (+ 236us) write_op.cc:489] Releasing partition, row and schema locks
0810 08:00:32.420983 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":560,"reported_tablets":4,"spinlock_wait_cycles":128066944,"child_traces":[["op",{"apply.queue_time_us":367,"apply.run_cpu_time_us":1641,"apply.run_wall_time_us":1641,"num_ops":1,"prepare.queue_time_us":66120,"prepare.run_cpu_time_us":3030,"prepare.run_wall_time_us":109804,"raft.queue_time_us":423,"raft.run_cpu_time_us":746,"raft.run_wall_time_us":881,"replication_time_us":34814,"spinlock_wait_cycles":36756224}]]}
I20260810 08:00:32.948115 15454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/master-2/data/info.pb
I20260810 08:00:32.950270 14960 tablet_service.cc:1514] Processing DeleteTablet for tablet 114d3eac994f466bb1787e066de05e2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:25 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:32.951114 15439 tablet_bootstrap.cc:492] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:32.954775 14960 tablet_service.cc:1514] Processing DeleteTablet for tablet 728f8f80b3bb4b10871d8b400f25377b with delete_type TABLET_DATA_DELETED (replaced by tablet e13b91d61063446f9d1454da4f8bb706) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:32.957280 14567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzwAODw/build/tsan/bin/kudu as pid 15454
I20260810 08:00:32.955389 15111 tablet_service.cc:1467] Processing CreateTablet for tablet 521d16ada2424d3aa41f838a511131fb (DEFAULT_TABLE table=default.Table_6cbd44cdd6a44bc4b3eb991d874353eb [id=5e6a4db5ec064eac935634a179729b1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:32.957098 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d295305b5a843eaa84c8ad666355b84" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:32.958891 15111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 521d16ada2424d3aa41f838a511131fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:32.960078 14918 leader_election.cc:302] T 2d295305b5a843eaa84c8ad666355b84 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a91b27762182454baada71e91ceab224; no voters: 3fc5e568f3724f158b616c303efb9eae, 9d7219686f304f62a54acc7497cbbd07
I20260810 08:00:32.961134 14960 tablet_service.cc:1514] Processing DeleteTablet for tablet d22a778bfe784b2f8a7b4d2216909f10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:26 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:32.961366 15831 raft_consensus.cc:3037] T 2d295305b5a843eaa84c8ad666355b84 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
W20260810 08:00:32.968223 14692 catalog_manager.cc:5687] Failed to send AlterTable RPC for tablet b5a0e0af742340b2945be2102768b307 (table default.Table_eab0fdadaf49486ca724a0a6e179a889 [id=fc52022e20044cc3bc88ac143e088e69], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet b5a0e0af742340b2945be2102768b307 consensus state has no leader
W20260810 08:00:32.971592 14681 catalog_manager.cc:4821] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet 3281dd757ff44d9498ef70ad372eb150 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3281dd757ff44d9498ef70ad372eb150 already in progress: creating tablet
I20260810 08:00:32.972636 14964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 521d16ada2424d3aa41f838a511131fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:32.972944 15490 tablet_bootstrap.cc:654] T 8a8454f693494ab3b82f5b85c2d62bb2 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:32.972985 15831 raft_consensus.cc:2726] T 2d295305b5a843eaa84c8ad666355b84 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 08:00:32.984310 15076 leader_election.cc:341] T 561cbc6a4eb14d3584f88ea26a0fea7a P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:32.984766 15076 leader_election.cc:302] T 561cbc6a4eb14d3584f88ea26a0fea7a P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:32.984999 15112 tablet_service.cc:1514] Processing DeleteTablet for tablet 4381ccb8711744ff8f9766ef49c53e28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:31 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:32.985882 15795 raft_consensus.cc:2726] T 561cbc6a4eb14d3584f88ea26a0fea7a P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 08:00:32.987572 15234 leader_election.cc:341] T 2cca8a18778c49d0be01363c31ef180b P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Not found: Tablet not found: 2cca8a18778c49d0be01363c31ef180b
I20260810 08:00:32.987962 15112 tablet_service.cc:1514] Processing DeleteTablet for tablet cb37a3a87f8b48a68c19510f306073c2 with delete_type TABLET_DATA_DELETED (replaced by tablet b18980bffe6045b5b69bcecb47b4c711) from {username='slave'} at 127.0.0.1:43002
W20260810 08:00:32.988411 15234 leader_election.cc:341] T d0761b090d8147b8a5537b0ed51108a9 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Not found: Tablet not found: d0761b090d8147b8a5537b0ed51108a9
I20260810 08:00:32.994285 15275 tablet_service.cc:1467] Processing CreateTablet for tablet 521d16ada2424d3aa41f838a511131fb (DEFAULT_TABLE table=default.Table_6cbd44cdd6a44bc4b3eb991d874353eb [id=5e6a4db5ec064eac935634a179729b1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:32.995414 15275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 521d16ada2424d3aa41f838a511131fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:33.001837 15831 raft_consensus.cc:482] T 25de7bf64bf6450ba17b7748568bc0ed P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:33.002552 15831 raft_consensus.cc:504] T 25de7bf64bf6450ba17b7748568bc0ed P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:33.006317 15831 leader_election.cc:288] T 25de7bf64bf6450ba17b7748568bc0ed P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:33.013980 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25de7bf64bf6450ba17b7748568bc0ed" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:33.016481 15439 tablet_bootstrap.cc:654] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:33.018879 15145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25de7bf64bf6450ba17b7748568bc0ed" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:33.019677 15145 raft_consensus.cc:2379] T 25de7bf64bf6450ba17b7748568bc0ed P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a91b27762182454baada71e91ceab224 in current term 1: Already voted for candidate 3fc5e568f3724f158b616c303efb9eae in this term.
I20260810 08:00:33.023886 15112 tablet_service.cc:1514] Processing DeleteTablet for tablet a1c4f8f514c94128aedd07bd5efdc3be with delete_type TABLET_DATA_DELETED (replaced by tablet a4a78b2a960d4d78aa1f48a9c9edf3be) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:33.035571 15112 tablet_service.cc:1514] Processing DeleteTablet for tablet 8364045e3fa2441e8dc1a1d87f73361d with delete_type TABLET_DATA_DELETED (replaced by tablet fd97cbfb16bf4ee2b4eed1e8bbbc7224) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:33.035930 15278 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cca8a18778c49d0be01363c31ef180b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:26 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:33.037010 15278 tablet_service.cc:1514] Processing DeleteTablet for tablet d22a778bfe784b2f8a7b4d2216909f10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:26 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:33.038817 15112 tablet_service.cc:1514] Processing DeleteTablet for tablet adada21bc1bf4f19a6347d0935bd74bd with delete_type TABLET_DATA_DELETED (replaced by tablet d25928e4c39d45e69c3efeb36f03aece) from {username='slave'} at 127.0.0.1:43002
W20260810 08:00:33.040645 14681 catalog_manager.cc:4808] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): delete failed for tablet 4381ccb8711744ff8f9766ef49c53e28 because the tablet was not found. No further retry: Not found: Tablet not found: 4381ccb8711744ff8f9766ef49c53e28
W20260810 08:00:32.970573 14692 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.14.57.194:55587 (request call id 49) took 909 ms (client timeout 1000 ms). Trace:
I20260810 08:00:33.048444 14686 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_043fe23b225140a7848446157e71c132"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:33.051023 14686 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_043fe23b225140a7848446157e71c132 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:33.055776 14690 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_eab0fdadaf49486ca724a0a6e179a889" } new_table_name: "default.Table_8642a50d2ed24720a1912ec9ea2e2939" modify_external_catalogs: true
I20260810 08:00:33.021112 14922 leader_election.cc:302] T 25de7bf64bf6450ba17b7748568bc0ed P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a91b27762182454baada71e91ceab224; no voters: 3fc5e568f3724f158b616c303efb9eae, 9d7219686f304f62a54acc7497cbbd07
I20260810 08:00:33.055109 14688 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_780441b622754f37a2cb73df2f19c3a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:33.060416 14688 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_780441b622754f37a2cb73df2f19c3a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:33.069399 15857 sys_catalog.cc:263] Verifying existing consensus state
I20260810 08:00:33.072901 15834 raft_consensus.cc:3037] T 25de7bf64bf6450ba17b7748568bc0ed P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
W20260810 08:00:33.079972 14689 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 68bc39707f44457fbf57c217ad827348 consensus state has no leader
W20260810 08:00:33.081480 14681 catalog_manager.cc:4808] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): delete failed for tablet cb37a3a87f8b48a68c19510f306073c2 because the tablet was not found. No further retry: Not found: Tablet not found: cb37a3a87f8b48a68c19510f306073c2
W20260810 08:00:33.082064 15234 consensus_peers.cc:487] T fbf62f4eab8b49f282df3ab4d95d886f P 3fc5e568f3724f158b616c303efb9eae -> Peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Couldn't send request to peer 9d7219686f304f62a54acc7497cbbd07. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fbf62f4eab8b49f282df3ab4d95d886f. This is attempt 1: this message will repeat every 5th retry.
I20260810 08:00:33.082758 15478 ts_tablet_manager.cc:1918] T 37d8a37a16f34a0285c43c0d71b5f072 P 3fc5e568f3724f158b616c303efb9eae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20260810 08:00:33.082893 14681 catalog_manager.cc:4808] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): delete failed for tablet a1c4f8f514c94128aedd07bd5efdc3be because the tablet was not found. No further retry: Not found: Tablet not found: a1c4f8f514c94128aedd07bd5efdc3be
I20260810 08:00:33.083127 15478 log.cc:1196] T 37d8a37a16f34a0285c43c0d71b5f072 P 3fc5e568f3724f158b616c303efb9eae: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal/wals/37d8a37a16f34a0285c43c0d71b5f072
I20260810 08:00:33.083632 15490 tablet_bootstrap.cc:492] T 8a8454f693494ab3b82f5b85c2d62bb2 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:33.083774 15478 ts_tablet_manager.cc:1939] T 37d8a37a16f34a0285c43c0d71b5f072 P 3fc5e568f3724f158b616c303efb9eae: Deleting consensus metadata
W20260810 08:00:33.083648 14681 catalog_manager.cc:4808] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): delete failed for tablet 8364045e3fa2441e8dc1a1d87f73361d because the tablet was not found. No further retry: Not found: Tablet not found: 8364045e3fa2441e8dc1a1d87f73361d
I20260810 08:00:33.084060 15490 ts_tablet_manager.cc:1397] T 8a8454f693494ab3b82f5b85c2d62bb2 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.150s user 0.008s sys 0.004s
W20260810 08:00:33.084705 14681 catalog_manager.cc:4808] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): delete failed for tablet adada21bc1bf4f19a6347d0935bd74bd because the tablet was not found. No further retry: Not found: Tablet not found: adada21bc1bf4f19a6347d0935bd74bd
I20260810 08:00:33.086374 15490 raft_consensus.cc:348] T 8a8454f693494ab3b82f5b85c2d62bb2 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:33.087064 15490 raft_consensus.cc:374] T 8a8454f693494ab3b82f5b85c2d62bb2 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:33.087307 15490 raft_consensus.cc:729] T 8a8454f693494ab3b82f5b85c2d62bb2 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:33.087975 15490 consensus_queue.cc:260] T 8a8454f693494ab3b82f5b85c2d62bb2 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
W20260810 08:00:33.089040 14681 catalog_manager.cc:4808] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet 114d3eac994f466bb1787e066de05e2c because the tablet was not found. No further retry: Not found: Tablet not found: 114d3eac994f466bb1787e066de05e2c
I20260810 08:00:33.095817 15490 ts_tablet_manager.cc:1428] T 8a8454f693494ab3b82f5b85c2d62bb2 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
W20260810 08:00:33.096287 14681 catalog_manager.cc:4808] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet 728f8f80b3bb4b10871d8b400f25377b because the tablet was not found. No further retry: Not found: Tablet not found: 728f8f80b3bb4b10871d8b400f25377b
I20260810 08:00:33.096683 15490 tablet_bootstrap.cc:492] T 389b30cba2b149e9ab0de8c191808db2 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
W20260810 08:00:33.097213 14681 catalog_manager.cc:4808] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet d22a778bfe784b2f8a7b4d2216909f10 because the tablet was not found. No further retry: Not found: Tablet not found: d22a778bfe784b2f8a7b4d2216909f10
W20260810 08:00:33.047053 14692 rpcz_store.cc:269] 0810 08:00:32.060822 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:32.060971 (+ 149us) service_pool.cc:227] Handling call
0810 08:00:32.126061 (+ 65090us) spinlock_profiling.cc:243] Waited 62.5 ms on lock 0x7b5400021554. stack: 00007fe2372c4033 00007fe2372c3f30 00007fe236e45534 00000000004db2b4 00000000004db249 00007fe23e1d0af0 00007fe23e253f2a 00007fe24011fd6c 00007fe240219c3e 00007fe23dd50fb4 00007fe23dd50f42 00007fe23dd50e71 00007fe23dd50dec 00007fe23dd500a2 00007fe23bffa94c 00007fe23bff9d86
0810 08:00:32.970506 (+844445us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":67063,"reported_tablets":11,"spinlock_wait_cycles":143681408}
I20260810 08:00:33.117699 15795 raft_consensus.cc:482] T 3ac554bd37b045a2a8cf2743f4645f28 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:33.118178 15795 raft_consensus.cc:504] T 3ac554bd37b045a2a8cf2743f4645f28 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:33.120024 15795 leader_election.cc:288] T 3ac554bd37b045a2a8cf2743f4645f28 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:33.122687 14695 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_93c38e9ecbdd4eeeb498ad5ebb5eef2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:33.124370 14986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ac554bd37b045a2a8cf2743f4645f28" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
W20260810 08:00:33.125811 15079 leader_election.cc:341] T 3ac554bd37b045a2a8cf2743f4645f28 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:33.129844 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ac554bd37b045a2a8cf2743f4645f28" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:33.130949 15783 tablet.cc:1722] T 49c35b7f4a72437fa0675abe8a8f3a71 P 3fc5e568f3724f158b616c303efb9eae: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260810 08:00:33.131271 15490 tablet_bootstrap.cc:654] T 389b30cba2b149e9ab0de8c191808db2 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:33.133512 15437 tablet_bootstrap.cc:492] T 924c0f73638f48a88c08d45d459132e1 P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:33.133953 15437 ts_tablet_manager.cc:1397] T 924c0f73638f48a88c08d45d459132e1 P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.230s user 0.005s sys 0.006s
I20260810 08:00:33.136139 15437 raft_consensus.cc:348] T 924c0f73638f48a88c08d45d459132e1 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:33.136785 15437 raft_consensus.cc:374] T 924c0f73638f48a88c08d45d459132e1 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:33.137027 15437 raft_consensus.cc:729] T 924c0f73638f48a88c08d45d459132e1 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:33.147501 15834 raft_consensus.cc:2726] T 25de7bf64bf6450ba17b7748568bc0ed P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 08:00:33.150997 15076 leader_election.cc:341] T 3ac554bd37b045a2a8cf2743f4645f28 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:33.151451 15076 leader_election.cc:302] T 3ac554bd37b045a2a8cf2743f4645f28 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:33.152498 15795 raft_consensus.cc:2726] T 3ac554bd37b045a2a8cf2743f4645f28 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:33.152806 14694 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet 5990e7ea243843bb869de5dde3e6cd23
I20260810 08:00:33.152930 15437 consensus_queue.cc:260] T 924c0f73638f48a88c08d45d459132e1 P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:33.159103 14964 tablet_service.cc:1514] Processing DeleteTablet for tablet 5990e7ea243843bb869de5dde3e6cd23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:32 UTC) from {username='slave'} at 127.0.0.1:34630
W20260810 08:00:33.160826 14681 catalog_manager.cc:4821] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet 5990e7ea243843bb869de5dde3e6cd23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5990e7ea243843bb869de5dde3e6cd23 already in progress: creating tablet
I20260810 08:00:33.163445 15437 ts_tablet_manager.cc:1428] T 924c0f73638f48a88c08d45d459132e1 P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.029s user 0.005s sys 0.000s
I20260810 08:00:33.165895 15834 raft_consensus.cc:482] T cb37a3a87f8b48a68c19510f306073c2 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:33.166812 15834 raft_consensus.cc:504] T cb37a3a87f8b48a68c19510f306073c2 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:33.171018 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb37a3a87f8b48a68c19510f306073c2" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:33.171672 15834 leader_election.cc:288] T cb37a3a87f8b48a68c19510f306073c2 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
W20260810 08:00:33.173048 14922 leader_election.cc:341] T cb37a3a87f8b48a68c19510f306073c2 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Not found: Tablet not found: cb37a3a87f8b48a68c19510f306073c2
W20260810 08:00:33.173559 14918 leader_election.cc:341] T cb37a3a87f8b48a68c19510f306073c2 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:33.174079 14918 leader_election.cc:302] T cb37a3a87f8b48a68c19510f306073c2 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a91b27762182454baada71e91ceab224; no voters: 3fc5e568f3724f158b616c303efb9eae, 9d7219686f304f62a54acc7497cbbd07
W20260810 08:00:33.174736 14695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93c38e9ecbdd4eeeb498ad5ebb5eef2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:33.175467 15834 raft_consensus.cc:2726] T cb37a3a87f8b48a68c19510f306073c2 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:33.178303 15278 tablet_service.cc:1514] Processing DeleteTablet for tablet 5990e7ea243843bb869de5dde3e6cd23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:32 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:33.176229 15437 tablet_bootstrap.cc:492] T b5a0e0af742340b2945be2102768b307 P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:33.189958 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 122) took 1732 ms. Trace:
I20260810 08:00:33.191757 15478 rpcz_store.cc:276] 0810 08:00:31.457451 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:31.461906 (+ 4455us) service_pool.cc:227] Handling call
0810 08:00:32.674754 (+1212848us) ts_tablet_manager.cc:1145] Deleting tablet 37d8a37a16f34a0285c43c0d71b5f072
0810 08:00:32.674813 (+ 59us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:32.924149 (+249336us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:33.082536 (+158387us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:33.085596 (+ 3060us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1206996}
I20260810 08:00:33.192847 15112 tablet_service.cc:1514] Processing DeleteTablet for tablet 5990e7ea243843bb869de5dde3e6cd23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:32 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:33.194017 15479 tablet_replica.cc:330] T 5990e7ea243843bb869de5dde3e6cd23 P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:33.193714 14694 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_3aad4373b6464997880bb90d36fd1c6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:33.194727 15479 raft_consensus.cc:2229] T 5990e7ea243843bb869de5dde3e6cd23 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:33.195159 15479 raft_consensus.cc:2258] T 5990e7ea243843bb869de5dde3e6cd23 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:33.195338 14678 catalog_manager.cc:4841] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): tablet 37d8a37a16f34a0285c43c0d71b5f072 (table default.Table_bae5256f457944049d86e384fb3759d6 [id=dad0c6e295df4fdd9b2c27d2b19fbc5f]) successfully deleted
W20260810 08:00:33.196079 14694 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3aad4373b6464997880bb90d36fd1c6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:33.196813 15479 ts_tablet_manager.cc:1905] T 5990e7ea243843bb869de5dde3e6cd23 P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:33.204519 15478 tablet_replica.cc:330] T fbf62f4eab8b49f282df3ab4d95d886f P 3fc5e568f3724f158b616c303efb9eae: stopping tablet replica
I20260810 08:00:33.208835 14687 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet 68048e2768cf441d906e675254175c9c
I20260810 08:00:33.210487 14964 tablet_service.cc:1514] Processing DeleteTablet for tablet 68048e2768cf441d906e675254175c9c with delete_type TABLET_DATA_DELETED (replaced by tablet f221dc79f1104fd0ab225c09da0df457) from {username='slave'} at 127.0.0.1:34630
W20260810 08:00:33.212080 14681 catalog_manager.cc:4821] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet 68048e2768cf441d906e675254175c9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68048e2768cf441d906e675254175c9c already in progress: creating tablet
I20260810 08:00:33.222357 14693 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet 8a0e1ad5653f46b4861b3e39629fbd2c
I20260810 08:00:33.223348 15439 tablet_bootstrap.cc:492] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:33.223749 15439 ts_tablet_manager.cc:1397] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.273s user 0.008s sys 0.002s
I20260810 08:00:33.224577 14693 master_service.cc:946] ReplaceTablet: received request to replace tablet 461de46896ee492b9f7cd1202208c19a from {username='slave'} at 127.0.0.1:42300
I20260810 08:00:33.224629 14964 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a0e1ad5653f46b4861b3e39629fbd2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:32 UTC) from {username='slave'} at 127.0.0.1:34630
W20260810 08:00:33.228264 14681 catalog_manager.cc:4821] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet 8a0e1ad5653f46b4861b3e39629fbd2c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a0e1ad5653f46b4861b3e39629fbd2c already in progress: creating tablet
I20260810 08:00:33.229578 15112 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a0e1ad5653f46b4861b3e39629fbd2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:32 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:33.231338 15439 raft_consensus.cc:348] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:33.232235 15439 raft_consensus.cc:374] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:33.232512 15439 raft_consensus.cc:729] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:33.233083 14687 catalog_manager.cc:6565] ReplaceTablet: tablet 68048e2768cf441d906e675254175c9c deleted and replaced by tablet f221dc79f1104fd0ab225c09da0df457
I20260810 08:00:33.234287 15112 tablet_service.cc:1514] Processing DeleteTablet for tablet 68048e2768cf441d906e675254175c9c with delete_type TABLET_DATA_DELETED (replaced by tablet f221dc79f1104fd0ab225c09da0df457) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:33.235507 15354 heartbeater.cc:347] Connected to a master server at 127.14.57.252:33819
I20260810 08:00:33.236445 15354 heartbeater.cc:503] Master 127.14.57.252:33819 was elected leader, sending a full tablet report...
I20260810 08:00:33.238997 15870 tablet.cc:1722] T 49c35b7f4a72437fa0675abe8a8f3a71 P 9d7219686f304f62a54acc7497cbbd07: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260810 08:00:33.241873 15275 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a0e1ad5653f46b4861b3e39629fbd2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:32 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:33.242228 15278 tablet_service.cc:1514] Processing DeleteTablet for tablet 68048e2768cf441d906e675254175c9c with delete_type TABLET_DATA_DELETED (replaced by tablet f221dc79f1104fd0ab225c09da0df457) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:33.245807 15478 raft_consensus.cc:2229] T fbf62f4eab8b49f282df3ab4d95d886f P 3fc5e568f3724f158b616c303efb9eae [term 1 LEADER]: Raft consensus shutting down.
I20260810 08:00:33.246562 15478 pending_rounds.cc:62] T fbf62f4eab8b49f282df3ab4d95d886f P 3fc5e568f3724f158b616c303efb9eae: Trying to abort 1 pending ops.
I20260810 08:00:33.247020 15478 pending_rounds.cc:68] T fbf62f4eab8b49f282df3ab4d95d886f P 3fc5e568f3724f158b616c303efb9eae: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316884812636995584 op_type: NO_OP noop_request { }
I20260810 08:00:33.247430 15478 raft_consensus.cc:2866] T fbf62f4eab8b49f282df3ab4d95d886f P 3fc5e568f3724f158b616c303efb9eae [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260810 08:00:33.247713 15478 raft_consensus.cc:2258] T fbf62f4eab8b49f282df3ab4d95d886f P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:33.253665 15871 tablet.cc:1722] T 49c35b7f4a72437fa0675abe8a8f3a71 P a91b27762182454baada71e91ceab224: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260810 08:00:33.257751 15478 ts_tablet_manager.cc:1905] T fbf62f4eab8b49f282df3ab4d95d886f P 3fc5e568f3724f158b616c303efb9eae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:33.262526 15857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be: Bootstrap starting.
I20260810 08:00:33.265807 15439 consensus_queue.cc:260] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:33.272266 15437 tablet_bootstrap.cc:654] T b5a0e0af742340b2945be2102768b307 P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:33.282327 15479 ts_tablet_manager.cc:1918] T 5990e7ea243843bb869de5dde3e6cd23 P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:33.282758 15479 log.cc:1196] T 5990e7ea243843bb869de5dde3e6cd23 P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/5990e7ea243843bb869de5dde3e6cd23
I20260810 08:00:33.283425 15479 ts_tablet_manager.cc:1939] T 5990e7ea243843bb869de5dde3e6cd23 P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:33.285987 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet 5990e7ea243843bb869de5dde3e6cd23 (table default.Table_870c8e79c4574433bf1b38ff75564dc9 [id=3def70102e974890b92f64b5c0ff308d]) successfully deleted
I20260810 08:00:33.291893 15199 heartbeater.cc:347] Connected to a master server at 127.14.57.252:33819
I20260810 08:00:33.300967 15439 ts_tablet_manager.cc:1428] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.077s user 0.009s sys 0.000s
I20260810 08:00:33.303725 14688 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_6cbd44cdd6a44bc4b3eb991d874353eb" } new_table_name: "default.Table_de9a90e3e5d841bd97301a0672436fdf" modify_external_catalogs: true
I20260810 08:00:33.316601 15039 heartbeater.cc:347] Connected to a master server at 127.14.57.252:33819
I20260810 08:00:33.324281 15439 tablet_bootstrap.cc:492] T 4381ccb8711744ff8f9766ef49c53e28 P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:33.331555 15437 tablet_bootstrap.cc:492] T b5a0e0af742340b2945be2102768b307 P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:33.354095 15437 ts_tablet_manager.cc:1397] T b5a0e0af742340b2945be2102768b307 P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.181s user 0.008s sys 0.011s
I20260810 08:00:33.365808 15490 tablet_bootstrap.cc:492] T 389b30cba2b149e9ab0de8c191808db2 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:33.368328 15490 ts_tablet_manager.cc:1397] T 389b30cba2b149e9ab0de8c191808db2 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.272s user 0.011s sys 0.001s
I20260810 08:00:33.376319 15490 raft_consensus.cc:348] T 389b30cba2b149e9ab0de8c191808db2 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:33.378329 15490 raft_consensus.cc:374] T 389b30cba2b149e9ab0de8c191808db2 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:33.386361 15818 raft_consensus.cc:482] T 68bc39707f44457fbf57c217ad827348 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:33.387012 15490 raft_consensus.cc:729] T 389b30cba2b149e9ab0de8c191808db2 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:33.387089 15818 raft_consensus.cc:504] T 68bc39707f44457fbf57c217ad827348 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:33.387960 15490 consensus_queue.cc:260] T 389b30cba2b149e9ab0de8c191808db2 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:33.392081 15818 leader_election.cc:288] T 68bc39707f44457fbf57c217ad827348 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:33.395437 15146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68bc39707f44457fbf57c217ad827348" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:33.395469 14986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68bc39707f44457fbf57c217ad827348" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:33.396133 15146 raft_consensus.cc:2454] T 68bc39707f44457fbf57c217ad827348 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 0.
W20260810 08:00:33.397176 15234 leader_election.cc:341] T 68bc39707f44457fbf57c217ad827348 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:33.397868 15234 leader_election.cc:302] T 68bc39707f44457fbf57c217ad827348 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, 9d7219686f304f62a54acc7497cbbd07; no voters: a91b27762182454baada71e91ceab224
I20260810 08:00:33.399065 15818 raft_consensus.cc:2781] T 68bc39707f44457fbf57c217ad827348 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:33.399281 15490 ts_tablet_manager.cc:1428] T 389b30cba2b149e9ab0de8c191808db2 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.031s user 0.006s sys 0.000s
I20260810 08:00:33.399667 15818 raft_consensus.cc:482] T 68bc39707f44457fbf57c217ad827348 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:33.400269 15490 tablet_bootstrap.cc:492] T e13b91d61063446f9d1454da4f8bb706 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:33.400467 14694 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_36a773333abe4fcbaf61c6209c63f796" } modify_external_catalogs: true
I20260810 08:00:33.400261 15818 raft_consensus.cc:3037] T 68bc39707f44457fbf57c217ad827348 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:33.401093 14694 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_36a773333abe4fcbaf61c6209c63f796" } modify_external_catalogs: true
I20260810 08:00:33.406190 15490 tablet_bootstrap.cc:654] T e13b91d61063446f9d1454da4f8bb706 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:33.356781 15437 raft_consensus.cc:348] T b5a0e0af742340b2945be2102768b307 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:33.410097 15437 raft_consensus.cc:374] T b5a0e0af742340b2945be2102768b307 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:33.410521 15437 raft_consensus.cc:729] T b5a0e0af742340b2945be2102768b307 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:33.411305 15437 consensus_queue.cc:260] T b5a0e0af742340b2945be2102768b307 P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:33.414278 15437 ts_tablet_manager.cc:1428] T b5a0e0af742340b2945be2102768b307 P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.060s user 0.004s sys 0.002s
I20260810 08:00:33.430320 14695 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_fea0f565d8a0450a91e8729141f89d99" } modify_external_catalogs: true
I20260810 08:00:33.430877 14695 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_fea0f565d8a0450a91e8729141f89d99" } modify_external_catalogs: true
I20260810 08:00:33.434738 15439 tablet_bootstrap.cc:654] T 4381ccb8711744ff8f9766ef49c53e28 P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
W20260810 08:00:33.441661 14688 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 521d16ada2424d3aa41f838a511131fb consensus state has no leader
W20260810 08:00:33.448789 14681 catalog_manager.cc:4821] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): delete failed for tablet 8a0e1ad5653f46b4861b3e39629fbd2c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a0e1ad5653f46b4861b3e39629fbd2c already in progress: creating tablet
W20260810 08:00:33.449549 14681 catalog_manager.cc:4821] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): delete failed for tablet 68048e2768cf441d906e675254175c9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68048e2768cf441d906e675254175c9c already in progress: creating tablet
I20260810 08:00:33.454454 14688 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_42d526ee0aeb4b6b8b7e99b05d6b67d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:33.456058 14688 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42d526ee0aeb4b6b8b7e99b05d6b67d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:33.461074 15437 tablet_bootstrap.cc:492] T 1f427f4093e5417a9ca79b1805b0675e P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:33.451202 15813 consensus_queue.cc:1035] T 2d295305b5a843eaa84c8ad666355b84 P 3fc5e568f3724f158b616c303efb9eae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 08:00:33.489986 15813 raft_consensus.cc:482] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:33.490823 15813 raft_consensus.cc:504] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:33.496853 14986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8bb1c44a7b4d17bddeff8b2fe81b35" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:33.497458 14986 raft_consensus.cc:2454] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 0.
I20260810 08:00:33.498653 15234 leader_election.cc:302] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224; no voters:
I20260810 08:00:33.499855 15813 leader_election.cc:288] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:33.501041 14688 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_7ed048f56dd943beb001f4add1ee3332"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:33.502625 15813 raft_consensus.cc:2781] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:33.507555 15813 raft_consensus.cc:482] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260810 08:00:33.508802 14688 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ed048f56dd943beb001f4add1ee3332 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:33.509579 15813 raft_consensus.cc:3037] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:33.516682 15834 raft_consensus.cc:482] T 561cbc6a4eb14d3584f88ea26a0fea7a P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:33.517170 14686 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_e3bf1bdea6324b319c1a04b7231674f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:33.517323 15834 raft_consensus.cc:504] T 561cbc6a4eb14d3584f88ea26a0fea7a P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
W20260810 08:00:33.519169 14686 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3bf1bdea6324b319c1a04b7231674f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:33.519975 15490 tablet_bootstrap.cc:492] T e13b91d61063446f9d1454da4f8bb706 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:33.520494 15490 ts_tablet_manager.cc:1397] T e13b91d61063446f9d1454da4f8bb706 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.120s user 0.013s sys 0.000s
I20260810 08:00:33.524709 15145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561cbc6a4eb14d3584f88ea26a0fea7a" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:33.525842 15145 raft_consensus.cc:2454] T 561cbc6a4eb14d3584f88ea26a0fea7a P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 0.
I20260810 08:00:33.527154 15834 leader_election.cc:288] T 561cbc6a4eb14d3584f88ea26a0fea7a P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:33.526367 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561cbc6a4eb14d3584f88ea26a0fea7a" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:33.528353 14922 leader_election.cc:302] T 561cbc6a4eb14d3584f88ea26a0fea7a P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224; no voters:
I20260810 08:00:33.529584 15834 raft_consensus.cc:2781] T 561cbc6a4eb14d3584f88ea26a0fea7a P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260810 08:00:33.529675 14918 leader_election.cc:341] T 561cbc6a4eb14d3584f88ea26a0fea7a P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:33.530248 15834 raft_consensus.cc:482] T 561cbc6a4eb14d3584f88ea26a0fea7a P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:33.530682 15834 raft_consensus.cc:3037] T 561cbc6a4eb14d3584f88ea26a0fea7a P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:33.546818 15818 raft_consensus.cc:504] T 68bc39707f44457fbf57c217ad827348 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:33.546294 14686 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_d48d3754e2e84c28b58869ee05f4ea95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:33.548081 15490 raft_consensus.cc:348] T e13b91d61063446f9d1454da4f8bb706 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:33.548182 15834 raft_consensus.cc:504] T 561cbc6a4eb14d3584f88ea26a0fea7a P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:33.548796 15490 raft_consensus.cc:374] T e13b91d61063446f9d1454da4f8bb706 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:33.549052 15490 raft_consensus.cc:729] T e13b91d61063446f9d1454da4f8bb706 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:33.553323 15145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561cbc6a4eb14d3584f88ea26a0fea7a" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07"
I20260810 08:00:33.553905 15145 raft_consensus.cc:3037] T 561cbc6a4eb14d3584f88ea26a0fea7a P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Advancing to term 1
W20260810 08:00:33.554936 14686 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d48d3754e2e84c28b58869ee05f4ea95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:33.560756 15795 raft_consensus.cc:482] T 8a8454f693494ab3b82f5b85c2d62bb2 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:33.561254 15795 raft_consensus.cc:504] T 8a8454f693494ab3b82f5b85c2d62bb2 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:33.562309 15490 consensus_queue.cc:260] T e13b91d61063446f9d1454da4f8bb706 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:33.563303 15795 leader_election.cc:288] T 8a8454f693494ab3b82f5b85c2d62bb2 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:33.569121 14688 master_service.cc:946] ReplaceTablet: received request to replace tablet a92d01d83dbf4b6ba47464dcefa8b2ae from {username='slave'} at 127.0.0.1:42300
I20260810 08:00:33.573895 14986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8454f693494ab3b82f5b85c2d62bb2" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
W20260810 08:00:33.575826 15079 leader_election.cc:341] T 8a8454f693494ab3b82f5b85c2d62bb2 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:33.578784 15145 raft_consensus.cc:2454] T 561cbc6a4eb14d3584f88ea26a0fea7a P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 1.
I20260810 08:00:33.580101 14922 leader_election.cc:302] T 561cbc6a4eb14d3584f88ea26a0fea7a P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224; no voters:
I20260810 08:00:33.581344 15834 leader_election.cc:288] T 561cbc6a4eb14d3584f88ea26a0fea7a P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Requested vote from peers 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:33.581846 15831 raft_consensus.cc:2781] T 561cbc6a4eb14d3584f88ea26a0fea7a P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 08:00:33.582410 15831 raft_consensus.cc:686] T 561cbc6a4eb14d3584f88ea26a0fea7a P a91b27762182454baada71e91ceab224 [term 1 LEADER]: Becoming Leader. State: Replica: a91b27762182454baada71e91ceab224, State: Running, Role: LEADER
I20260810 08:00:33.583329 15831 consensus_queue.cc:237] T 561cbc6a4eb14d3584f88ea26a0fea7a P a91b27762182454baada71e91ceab224 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:33.586563 15795 raft_consensus.cc:482] T 8e603d04f7454501b2edfb5a1f4ea7f0 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:33.587054 15795 raft_consensus.cc:504] T 8e603d04f7454501b2edfb5a1f4ea7f0 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:33.598613 14986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e603d04f7454501b2edfb5a1f4ea7f0" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:33.599682 15795 leader_election.cc:288] T 8e603d04f7454501b2edfb5a1f4ea7f0 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:33.601989 15795 raft_consensus.cc:482] T e13b91d61063446f9d1454da4f8bb706 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:33.603565 15437 tablet_bootstrap.cc:654] T 1f427f4093e5417a9ca79b1805b0675e P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:33.476676 15478 ts_tablet_manager.cc:1918] T fbf62f4eab8b49f282df3ab4d95d886f P 3fc5e568f3724f158b616c303efb9eae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 08:00:33.610782 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561cbc6a4eb14d3584f88ea26a0fea7a" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae"
I20260810 08:00:33.619923 15478 log.cc:1196] T fbf62f4eab8b49f282df3ab4d95d886f P 3fc5e568f3724f158b616c303efb9eae: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal/wals/fbf62f4eab8b49f282df3ab4d95d886f
I20260810 08:00:33.620860 15478 ts_tablet_manager.cc:1939] T fbf62f4eab8b49f282df3ab4d95d886f P 3fc5e568f3724f158b616c303efb9eae: Deleting consensus metadata
I20260810 08:00:33.623945 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 123) took 2161 ms. Trace:
W20260810 08:00:33.602716 15079 leader_election.cc:341] T 8e603d04f7454501b2edfb5a1f4ea7f0 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:33.625845 15795 raft_consensus.cc:504] T e13b91d61063446f9d1454da4f8bb706 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:33.634768 15478 rpcz_store.cc:276] 0810 08:00:31.462876 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:31.465892 (+ 3016us) service_pool.cc:227] Handling call
0810 08:00:33.194841 (+1728949us) ts_tablet_manager.cc:1145] Deleting tablet fbf62f4eab8b49f282df3ab4d95d886f
0810 08:00:33.194894 (+ 53us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:33.344997 (+150103us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:33.476381 (+131384us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:33.623880 (+147499us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38197,"tablet-delete.queue_time_us":1726703}
I20260810 08:00:33.642061 14692 catalog_manager.cc:5587] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265) does not have the latest schema for tablet 579bac622ddf40bf80d01d66ae2fa100 (table default.Table_02bc67e213c241fabead65c8dc785aa8 [id=650bc0c97e7842798d575199f1ea662c]). Expected version 1 got 0
I20260810 08:00:33.643178 14692 catalog_manager.cc:5587] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265) does not have the latest schema for tablet b5a0e0af742340b2945be2102768b307 (table default.Table_eab0fdadaf49486ca724a0a6e179a889 [id=fc52022e20044cc3bc88ac143e088e69]). Expected version 2 got 0
I20260810 08:00:33.643967 14692 catalog_manager.cc:5587] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265) does not have the latest schema for tablet 49c35b7f4a72437fa0675abe8a8f3a71 (table default.Table_1567aaa1d2ab44028745e82c29232682 [id=6cf63d3e931a4f6487074494254805bb]). Expected version 4 got 0
I20260810 08:00:33.645207 14986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13b91d61063446f9d1454da4f8bb706" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
W20260810 08:00:33.646936 15079 leader_election.cc:341] T e13b91d61063446f9d1454da4f8bb706 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:33.648757 15478 tablet_replica.cc:330] T 728f8f80b3bb4b10871d8b400f25377b P 3fc5e568f3724f158b616c303efb9eae: stopping tablet replica
I20260810 08:00:33.649344 15490 ts_tablet_manager.cc:1428] T e13b91d61063446f9d1454da4f8bb706 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.129s user 0.009s sys 0.007s
I20260810 08:00:33.649253 15857 log.cc:824] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be: Log is configured to *not* fsync() on all Append() calls
I20260810 08:00:33.650321 15490 tablet_bootstrap.cc:492] T 3281dd757ff44d9498ef70ad372eb150 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:33.652197 15437 tablet_bootstrap.cc:492] T 1f427f4093e5417a9ca79b1805b0675e P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:33.653199 15478 raft_consensus.cc:2229] T 728f8f80b3bb4b10871d8b400f25377b P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Raft consensus shutting down.
W20260810 08:00:33.655202 14690 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b5a0e0af742340b2945be2102768b307 consensus state has no leader
W20260810 08:00:33.655198 14918 leader_election.cc:341] T 561cbc6a4eb14d3584f88ea26a0fea7a P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:33.655472 15437 ts_tablet_manager.cc:1397] T 1f427f4093e5417a9ca79b1805b0675e P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.195s user 0.010s sys 0.000s
I20260810 08:00:33.657182 15478 raft_consensus.cc:2258] T 728f8f80b3bb4b10871d8b400f25377b P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:33.659175 14691 catalog_manager.cc:5587] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601) does not have the latest schema for tablet 49c35b7f4a72437fa0675abe8a8f3a71 (table default.Table_1567aaa1d2ab44028745e82c29232682 [id=6cf63d3e931a4f6487074494254805bb]). Expected version 4 got 0
I20260810 08:00:33.660497 14691 catalog_manager.cc:5587] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601) does not have the latest schema for tablet 2d295305b5a843eaa84c8ad666355b84 (table default.Table_d5c08115c8b24ad48a9686e48a45ab82 [id=bb6d5450013a45b1bb6a85bc9e4e6d9c]). Expected version 1 got 0
W20260810 08:00:33.654062 15234 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.193:43601, user_credentials={real_user=slave}} blocked reactor thread for 155468us
I20260810 08:00:33.662426 14678 catalog_manager.cc:4841] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): tablet fbf62f4eab8b49f282df3ab4d95d886f (table default.Table_7f3a4e19ded64b3783a00b3d4c6839c3 [id=6c6afcf578534997878b98feb9574e2d]) successfully deleted
I20260810 08:00:33.663667 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8454f693494ab3b82f5b85c2d62bb2" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:33.667979 14964 tablet_service.cc:1467] Processing CreateTablet for tablet 8a30ecac20d040309b69ec5137a5a927 (DEFAULT_TABLE table=default.Table_5f5037d84e3f4dcc9d14d94a619e4d6a [id=f3ea459eae7f4e548181217282e69670]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:33.669344 15112 tablet_service.cc:1467] Processing CreateTablet for tablet 8a30ecac20d040309b69ec5137a5a927 (DEFAULT_TABLE table=default.Table_5f5037d84e3f4dcc9d14d94a619e4d6a [id=f3ea459eae7f4e548181217282e69670]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:33.670619 15112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a30ecac20d040309b69ec5137a5a927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:33.670709 15818 leader_election.cc:288] T 68bc39707f44457fbf57c217ad827348 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Requested vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:33.679150 15437 raft_consensus.cc:348] T 1f427f4093e5417a9ca79b1805b0675e P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:33.683204 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e603d04f7454501b2edfb5a1f4ea7f0" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:33.684003 15478 ts_tablet_manager.cc:1905] T 728f8f80b3bb4b10871d8b400f25377b P 3fc5e568f3724f158b616c303efb9eae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:33.684420 15881 raft_consensus.cc:482] T 579bac622ddf40bf80d01d66ae2fa100 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:33.684937 15881 raft_consensus.cc:504] T 579bac622ddf40bf80d01d66ae2fa100 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:33.691419 15881 leader_election.cc:288] T 579bac622ddf40bf80d01d66ae2fa100 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:33.698642 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13b91d61063446f9d1454da4f8bb706" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:33.701319 15813 raft_consensus.cc:504] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:33.703365 15813 leader_election.cc:288] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Requested vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:33.680027 14960 tablet_service.cc:1467] Processing CreateTablet for tablet 00930573a1c340ff82573e14b792a63c (DEFAULT_TABLE table=default.Table_27c2e400be3e44cd8d3f1738490182e6 [id=ca4fa21132484f9bbdfbcfbe92ab99a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:33.685290 14964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a30ecac20d040309b69ec5137a5a927. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 08:00:33.709969 15076 leader_election.cc:341] T 8a8454f693494ab3b82f5b85c2d62bb2 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:33.710390 15076 leader_election.cc:302] T 8a8454f693494ab3b82f5b85c2d62bb2 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:33.711669 14960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00930573a1c340ff82573e14b792a63c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:33.712327 14689 catalog_manager.cc:5495] T 25de7bf64bf6450ba17b7748568bc0ed P 3fc5e568f3724f158b616c303efb9eae reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fc5e568f3724f158b616c303efb9eae (127.14.57.195). New cstate: current_term: 1 leader_uuid: "3fc5e568f3724f158b616c303efb9eae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } health_report { overall_health: UNKNOWN } } }
I20260810 08:00:33.690557 14986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "579bac622ddf40bf80d01d66ae2fa100" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:33.714046 14689 catalog_manager.cc:5587] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729) does not have the latest schema for tablet 49c35b7f4a72437fa0675abe8a8f3a71 (table default.Table_1567aaa1d2ab44028745e82c29232682 [id=6cf63d3e931a4f6487074494254805bb]). Expected version 4 got 0
I20260810 08:00:33.714689 14689 catalog_manager.cc:5495] T 2d295305b5a843eaa84c8ad666355b84 P 3fc5e568f3724f158b616c303efb9eae reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fc5e568f3724f158b616c303efb9eae (127.14.57.195). New cstate: current_term: 1 leader_uuid: "3fc5e568f3724f158b616c303efb9eae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } health_report { overall_health: UNKNOWN } } }
I20260810 08:00:33.715865 14689 catalog_manager.cc:5587] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729) does not have the latest schema for tablet 2d295305b5a843eaa84c8ad666355b84 (table default.Table_d5c08115c8b24ad48a9686e48a45ab82 [id=bb6d5450013a45b1bb6a85bc9e4e6d9c]). Expected version 1 got 0
I20260810 08:00:33.716744 14689 catalog_manager.cc:5587] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729) does not have the latest schema for tablet 68bc39707f44457fbf57c217ad827348 (table default.Table_4f3c632faa1e4c2d86acf10a656ee7c7 [id=fbd03ee96b004797a503841f4cc7249a]). Expected version 1 got 0
W20260810 08:00:33.717185 15079 leader_election.cc:341] T 579bac622ddf40bf80d01d66ae2fa100 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:33.719125 14986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68bc39707f44457fbf57c217ad827348" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224"
I20260810 08:00:33.723819 14686 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_e6beb01e821446feaced3468aaa20563" } new_table_name: "default.Table_4f8046a53c72409ea8b0654a10d0f816" modify_external_catalogs: true
I20260810 08:00:33.724363 14695 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet 2fbda1b3b5b4482ba6f03f3e275d94e7
I20260810 08:00:33.728010 15111 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fbda1b3b5b4482ba6f03f3e275d94e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:33 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:33.728399 14967 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fbda1b3b5b4482ba6f03f3e275d94e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:33 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:33.728973 14690 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_5f6628b89c6a42e38f9e8e1b57979e48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:33.730242 14688 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet a92d01d83dbf4b6ba47464dcefa8b2ae
W20260810 08:00:33.731946 14690 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f6628b89c6a42e38f9e8e1b57979e48 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:33.727373 15146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68bc39707f44457fbf57c217ad827348" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07"
I20260810 08:00:33.732429 15146 raft_consensus.cc:3037] T 68bc39707f44457fbf57c217ad827348 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:33.732996 14953 tablet_service.cc:1514] Processing DeleteTablet for tablet a92d01d83dbf4b6ba47464dcefa8b2ae with delete_type TABLET_DATA_DELETED (replaced by tablet 0493eb96a2a042718527ddcd9524d799) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:33.723836 14693 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet 461de46896ee492b9f7cd1202208c19a
I20260810 08:00:33.745005 15881 raft_consensus.cc:2726] T 8a8454f693494ab3b82f5b85c2d62bb2 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:33.748150 15795 leader_election.cc:288] T e13b91d61063446f9d1454da4f8bb706 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:33.762621 15437 raft_consensus.cc:374] T 1f427f4093e5417a9ca79b1805b0675e P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:33.763029 15437 raft_consensus.cc:729] T 1f427f4093e5417a9ca79b1805b0675e P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:33.764822 15437 consensus_queue.cc:260] T 1f427f4093e5417a9ca79b1805b0675e P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
W20260810 08:00:33.766223 15076 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.195:37729, user_credentials={real_user=slave}} blocked reactor thread for 56285us
W20260810 08:00:33.767429 15076 leader_election.cc:341] T 8e603d04f7454501b2edfb5a1f4ea7f0 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:33.753489 15490 tablet_bootstrap.cc:654] T 3281dd757ff44d9498ef70ad372eb150 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:33.774309 14693 catalog_manager.cc:6565] ReplaceTablet: tablet 461de46896ee492b9f7cd1202208c19a deleted and replaced by tablet 1ce493c78a30404ca15e5b2568446bdf
I20260810 08:00:33.774214 14688 catalog_manager.cc:6565] ReplaceTablet: tablet a92d01d83dbf4b6ba47464dcefa8b2ae deleted and replaced by tablet 0493eb96a2a042718527ddcd9524d799
I20260810 08:00:33.775889 15146 raft_consensus.cc:2454] T 68bc39707f44457fbf57c217ad827348 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 1.
I20260810 08:00:33.775935 15275 tablet_service.cc:1467] Processing CreateTablet for tablet 8a30ecac20d040309b69ec5137a5a927 (DEFAULT_TABLE table=default.Table_5f5037d84e3f4dcc9d14d94a619e4d6a [id=f3ea459eae7f4e548181217282e69670]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:33.777175 15275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a30ecac20d040309b69ec5137a5a927. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 08:00:33.731628 14681 catalog_manager.cc:4821] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): delete failed for tablet 2fbda1b3b5b4482ba6f03f3e275d94e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2fbda1b3b5b4482ba6f03f3e275d94e7 already in progress: creating tablet
W20260810 08:00:33.784940 14681 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.14.57.194:36265, user_credentials={real_user=slave}} blocked reactor thread for 53350.6us
I20260810 08:00:33.786800 14967 tablet_service.cc:1514] Processing DeleteTablet for tablet 461de46896ee492b9f7cd1202208c19a with delete_type TABLET_DATA_DELETED (replaced by tablet 1ce493c78a30404ca15e5b2568446bdf) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:33.789125 15122 tablet_service.cc:1514] Processing DeleteTablet for tablet 461de46896ee492b9f7cd1202208c19a with delete_type TABLET_DATA_DELETED (replaced by tablet 1ce493c78a30404ca15e5b2568446bdf) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:33.768289 15076 leader_election.cc:302] T 8e603d04f7454501b2edfb5a1f4ea7f0 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:33.787706 15111 tablet_service.cc:1514] Processing DeleteTablet for tablet a92d01d83dbf4b6ba47464dcefa8b2ae with delete_type TABLET_DATA_DELETED (replaced by tablet 0493eb96a2a042718527ddcd9524d799) from {username='slave'} at 127.0.0.1:43002
W20260810 08:00:33.793427 14681 catalog_manager.cc:4821] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): delete failed for tablet 461de46896ee492b9f7cd1202208c19a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 461de46896ee492b9f7cd1202208c19a already in progress: creating tablet
W20260810 08:00:33.794139 14681 catalog_manager.cc:4821] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet 2fbda1b3b5b4482ba6f03f3e275d94e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2fbda1b3b5b4482ba6f03f3e275d94e7 already in progress: creating tablet
I20260810 08:00:33.794718 15889 raft_consensus.cc:2726] T 8e603d04f7454501b2edfb5a1f4ea7f0 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 08:00:33.794910 14681 catalog_manager.cc:4821] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): delete failed for tablet a92d01d83dbf4b6ba47464dcefa8b2ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a92d01d83dbf4b6ba47464dcefa8b2ae already in progress: creating tablet
I20260810 08:00:33.805425 15831 consensus_queue.cc:1035] T 2cca8a18778c49d0be01363c31ef180b P a91b27762182454baada71e91ceab224 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 08:00:33.810851 15437 ts_tablet_manager.cc:1428] T 1f427f4093e5417a9ca79b1805b0675e P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.155s user 0.006s sys 0.000s
I20260810 08:00:33.812951 15490 tablet_bootstrap.cc:492] T 3281dd757ff44d9498ef70ad372eb150 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:33.816087 15490 ts_tablet_manager.cc:1397] T 3281dd757ff44d9498ef70ad372eb150 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.166s user 0.007s sys 0.006s
I20260810 08:00:33.821406 15278 tablet_service.cc:1514] Processing DeleteTablet for tablet a92d01d83dbf4b6ba47464dcefa8b2ae with delete_type TABLET_DATA_DELETED (replaced by tablet 0493eb96a2a042718527ddcd9524d799) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:33.822643 15280 tablet_service.cc:1467] Processing CreateTablet for tablet 00930573a1c340ff82573e14b792a63c (DEFAULT_TABLE table=default.Table_27c2e400be3e44cd8d3f1738490182e6 [id=ca4fa21132484f9bbdfbcfbe92ab99a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:33.821080 14695 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_80fd70a445294836a1e5485e58cecde8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:33.824347 15280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00930573a1c340ff82573e14b792a63c. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 08:00:33.824518 14695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_80fd70a445294836a1e5485e58cecde8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:33.829418 14694 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet d3b41c4912f840c18913fe29c1af9fd9
I20260810 08:00:33.801772 15267 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fbda1b3b5b4482ba6f03f3e275d94e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:33 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:33.835685 14688 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_812e1f2d005b4c1a8edc57a1c48b7fa0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:33.837597 14688 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_812e1f2d005b4c1a8edc57a1c48b7fa0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:33.838881 15490 raft_consensus.cc:348] T 3281dd757ff44d9498ef70ad372eb150 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:33.839529 15490 raft_consensus.cc:374] T 3281dd757ff44d9498ef70ad372eb150 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:33.839766 15490 raft_consensus.cc:729] T 3281dd757ff44d9498ef70ad372eb150 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:33.843740 14693 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_c4fc713ca0b54050b0475b8b746658f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:33.845710 14693 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4fc713ca0b54050b0475b8b746658f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:33.848805 14967 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cca8a18778c49d0be01363c31ef180b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:26 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:33.849342 14953 tablet_service.cc:1514] Processing DeleteTablet for tablet d0761b090d8147b8a5537b0ed51108a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:27 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:33.850847 15439 tablet_bootstrap.cc:492] T 4381ccb8711744ff8f9766ef49c53e28 P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:33.850919 14953 tablet_service.cc:1514] Processing DeleteTablet for tablet cb37a3a87f8b48a68c19510f306073c2 with delete_type TABLET_DATA_DELETED (replaced by tablet b18980bffe6045b5b69bcecb47b4c711) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:33.851279 15439 ts_tablet_manager.cc:1397] T 4381ccb8711744ff8f9766ef49c53e28 P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.527s user 0.018s sys 0.015s
I20260810 08:00:33.852674 14953 tablet_service.cc:1514] Processing DeleteTablet for tablet cf8bb1c44a7b4d17bddeff8b2fe81b35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:30 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:33.854058 14953 tablet_service.cc:1514] Processing DeleteTablet for tablet d3b41c4912f840c18913fe29c1af9fd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:33 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:33.855615 15111 tablet_service.cc:1514] Processing DeleteTablet for tablet d3b41c4912f840c18913fe29c1af9fd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:33 UTC) from {username='slave'} at 127.0.0.1:43002
W20260810 08:00:33.856318 14691 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.14.57.193:35941 (request call id 37) took 792 ms (client timeout 1000 ms). Trace:
I20260810 08:00:33.856413 15490 consensus_queue.cc:260] T 3281dd757ff44d9498ef70ad372eb150 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:33.858559 15267 tablet_service.cc:1514] Processing DeleteTablet for tablet 461de46896ee492b9f7cd1202208c19a with delete_type TABLET_DATA_DELETED (replaced by tablet 1ce493c78a30404ca15e5b2568446bdf) from {username='slave'} at 127.0.0.1:40692
W20260810 08:00:33.859771 14681 catalog_manager.cc:4821] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet a92d01d83dbf4b6ba47464dcefa8b2ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a92d01d83dbf4b6ba47464dcefa8b2ae already in progress: creating tablet
W20260810 08:00:33.860445 15234 leader_election.cc:341] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Not found: Tablet not found: cf8bb1c44a7b4d17bddeff8b2fe81b35
I20260810 08:00:33.861721 15437 tablet_bootstrap.cc:492] T 68bc39707f44457fbf57c217ad827348 P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:33.861608 14694 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_d34e1afecbcf418483dec42d2f76f8d9" } new_table_name: "default.Table_fbd8cd0e547e4fdd8413c51f0f7da212" modify_external_catalogs: true
W20260810 08:00:33.862737 14679 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 0687d323788642f798465ece973702bb (table default.Table_c65aa418eb904b57aab18f8b403e8c8e [id=0d60b763c4804818ae26fa693a4fd808], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 0687d323788642f798465ece973702bb consensus state has no leader
I20260810 08:00:33.863584 15490 ts_tablet_manager.cc:1428] T 3281dd757ff44d9498ef70ad372eb150 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.047s user 0.001s sys 0.005s
I20260810 08:00:33.864447 15490 tablet_bootstrap.cc:492] T 4ca688b4031c41a9884829fbd0be1f33 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:33.864943 15831 raft_consensus.cc:482] T 924c0f73638f48a88c08d45d459132e1 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:33.865649 15831 raft_consensus.cc:504] T 924c0f73638f48a88c08d45d459132e1 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:33.854951 15439 raft_consensus.cc:348] T 4381ccb8711744ff8f9766ef49c53e28 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:33.868872 15439 raft_consensus.cc:374] T 4381ccb8711744ff8f9766ef49c53e28 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:33.869204 15439 raft_consensus.cc:729] T 4381ccb8711744ff8f9766ef49c53e28 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:33.869370 15831 leader_election.cc:288] T 924c0f73638f48a88c08d45d459132e1 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
W20260810 08:00:33.871696 14922 leader_election.cc:341] T 924c0f73638f48a88c08d45d459132e1 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Not found: Tablet not found: 924c0f73638f48a88c08d45d459132e1
I20260810 08:00:33.871815 15302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "924c0f73638f48a88c08d45d459132e1" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:33.872398 15439 consensus_queue.cc:260] T 4381ccb8711744ff8f9766ef49c53e28 P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
W20260810 08:00:33.856684 14691 rpcz_store.cc:269] 0810 08:00:33.063692 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:33.063836 (+ 144us) service_pool.cc:227] Handling call
0810 08:00:33.856247 (+792411us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":9}
W20260810 08:00:33.876194 14918 leader_election.cc:341] T 924c0f73638f48a88c08d45d459132e1 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:33.876808 14918 leader_election.cc:302] T 924c0f73638f48a88c08d45d459132e1 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a91b27762182454baada71e91ceab224; no voters: 3fc5e568f3724f158b616c303efb9eae, 9d7219686f304f62a54acc7497cbbd07
I20260810 08:00:33.881985 15831 raft_consensus.cc:2726] T 924c0f73638f48a88c08d45d459132e1 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 08:00:33.905690 14681 catalog_manager.cc:4808] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): delete failed for tablet d3b41c4912f840c18913fe29c1af9fd9 because the tablet was not found. No further retry: Not found: Tablet not found: d3b41c4912f840c18913fe29c1af9fd9
W20260810 08:00:33.906175 14681 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.14.57.194:36265, user_credentials={real_user=slave}} blocked reactor thread for 45146.5us
I20260810 08:00:33.908020 15111 tablet_service.cc:1467] Processing CreateTablet for tablet 00930573a1c340ff82573e14b792a63c (DEFAULT_TABLE table=default.Table_27c2e400be3e44cd8d3f1738490182e6 [id=ca4fa21132484f9bbdfbcfbe92ab99a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:33.909137 15111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00930573a1c340ff82573e14b792a63c. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 08:00:33.916199 15234 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.57.194:36265, user_credentials={real_user=slave}} blocked reactor thread for 51653.6us
W20260810 08:00:33.917032 14922 consensus_peers.cc:487] T 2cca8a18778c49d0be01363c31ef180b P a91b27762182454baada71e91ceab224 -> Peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Couldn't send request to peer 9d7219686f304f62a54acc7497cbbd07. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2cca8a18778c49d0be01363c31ef180b. This is attempt 1: this message will repeat every 5th retry.
I20260810 08:00:33.917451 15439 ts_tablet_manager.cc:1428] T 4381ccb8711744ff8f9766ef49c53e28 P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.066s user 0.006s sys 0.001s
I20260810 08:00:33.917788 15234 leader_election.cc:302] T 68bc39707f44457fbf57c217ad827348 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, 9d7219686f304f62a54acc7497cbbd07; no voters:
I20260810 08:00:33.918447 15818 raft_consensus.cc:2781] T 68bc39707f44457fbf57c217ad827348 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Leader election won for term 1
I20260810 08:00:33.919005 15818 raft_consensus.cc:686] T 68bc39707f44457fbf57c217ad827348 P 3fc5e568f3724f158b616c303efb9eae [term 1 LEADER]: Becoming Leader. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Running, Role: LEADER
W20260810 08:00:33.920028 14689 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.14.57.195:51883 (request call id 42) took 833 ms (client timeout 1000 ms). Trace:
I20260810 08:00:33.920334 15818 consensus_queue.cc:237] T 68bc39707f44457fbf57c217ad827348 P 3fc5e568f3724f158b616c303efb9eae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
W20260810 08:00:33.920476 14689 rpcz_store.cc:269] 0810 08:00:33.086212 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:33.086343 (+ 131us) service_pool.cc:227] Handling call
0810 08:00:33.100721 (+ 14378us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x7b5400021554. stack: 00007fe2372c4033 00007fe2372c3f30 00007fe236e45534 00000000004db2b4 00000000004db249 00007fe23e1d0af0 00007fe23e253f2a 00007fe24011fd6c 00007fe240219c3e 00007fe23dd50fb4 00007fe23dd50f42 00007fe23dd50e71 00007fe23dd50dec 00007fe23dd500a2 00007fe23bffa94c 00007fe23bff9d86
0810 08:00:33.919960 (+819239us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 08:00:33.850749 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 08:00:33.851110 (+ 361us) write_op.cc:432] Acquiring schema lock in shared mode
0810 08:00:33.851134 (+ 24us) write_op.cc:435] Acquired schema lock
0810 08:00:33.851147 (+ 13us) tablet.cc:660] Decoding operations
0810 08:00:33.851497 (+ 350us) write_op.cc:620] Acquiring the partition lock for write op
0810 08:00:33.851537 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0810 08:00:33.851560 (+ 23us) tablet.cc:683] Acquiring locks for 2 operations
0810 08:00:33.851881 (+ 321us) tablet.cc:699] Row locks acquired
0810 08:00:33.851895 (+ 14us) write_op.cc:260] PREPARE: finished
0810 08:00:33.852046 (+ 151us) write_op.cc:270] Start()
0810 08:00:33.852140 (+ 94us) write_op.cc:276] Timestamp: P: 1786348833851926 usec, L: 0
0810 08:00:33.852156 (+ 16us) op_driver.cc:331] REPLICATION: starting
0810 08:00:33.852720 (+ 564us) log.cc:842] Serialized 913 byte log entry
0810 08:00:33.912105 (+ 59385us) op_driver.cc:444] REPLICATION: finished
0810 08:00:33.912832 (+ 727us) write_op.cc:301] APPLY: starting
0810 08:00:33.912890 (+ 58us) tablet.cc:1365] starting BulkCheckPresence
0810 08:00:33.913022 (+ 132us) tablet.cc:1368] finished BulkCheckPresence
0810 08:00:33.913118 (+ 96us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 08:00:33.913368 (+ 250us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 08:00:33.913522 (+ 154us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0810 08:00:33.913564 (+ 42us) write_op.cc:312] APPLY: finished
0810 08:00:33.913769 (+ 205us) log.cc:842] Serialized 29 byte log entry
0810 08:00:33.913918 (+ 149us) write_op.cc:489] Releasing partition, row and schema locks
0810 08:00:33.913990 (+ 72us) write_op.cc:454] Released schema lock
0810 08:00:33.914026 (+ 36us) write_op.cc:341] FINISH: Updating metrics
0810 08:00:33.914215 (+ 189us) write_op.cc:489] Releasing partition, row and schema locks
0810 08:00:33.914238 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":6,"spinlock_wait_cycles":27756800,"child_traces":[["op",{"apply.queue_time_us":569,"apply.run_cpu_time_us":1580,"apply.run_wall_time_us":1580,"num_ops":2,"prepare.queue_time_us":117514,"prepare.run_cpu_time_us":2352,"prepare.run_wall_time_us":18205,"raft.queue_time_us":111,"raft.run_cpu_time_us":796,"raft.run_wall_time_us":786,"replication_time_us":59860}]]}
I20260810 08:00:33.926550 14691 catalog_manager.cc:5495] T 561cbc6a4eb14d3584f88ea26a0fea7a P a91b27762182454baada71e91ceab224 reported cstate change: term changed from 0 to 1, leader changed from <none> to a91b27762182454baada71e91ceab224 (127.14.57.193). New cstate: current_term: 1 leader_uuid: "a91b27762182454baada71e91ceab224" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } health_report { overall_health: UNKNOWN } } }
I20260810 08:00:33.928331 15890 tablet_replica.cc:330] T 2cca8a18778c49d0be01363c31ef180b P a91b27762182454baada71e91ceab224: stopping tablet replica
I20260810 08:00:33.934777 15122 tablet_service.cc:1514] Processing DeleteTablet for tablet 5990e7ea243843bb869de5dde3e6cd23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:32 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:33.946108 15889 raft_consensus.cc:482] T 389b30cba2b149e9ab0de8c191808db2 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:33.946939 15889 raft_consensus.cc:504] T 389b30cba2b149e9ab0de8c191808db2 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:33.948601 14691 catalog_manager.cc:5587] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601) does not have the latest schema for tablet b5a0e0af742340b2945be2102768b307 (table default.Table_8642a50d2ed24720a1912ec9ea2e2939 [id=fc52022e20044cc3bc88ac143e088e69]). Expected version 3 got 0
I20260810 08:00:33.951545 15889 leader_election.cc:288] T 389b30cba2b149e9ab0de8c191808db2 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
W20260810 08:00:33.944787 14681 catalog_manager.cc:4808] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): delete failed for tablet 5990e7ea243843bb869de5dde3e6cd23 because the tablet was not found. No further retry: Not found: Tablet not found: 5990e7ea243843bb869de5dde3e6cd23
W20260810 08:00:33.969115 14681 catalog_manager.cc:4821] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet 461de46896ee492b9f7cd1202208c19a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 461de46896ee492b9f7cd1202208c19a already in progress: creating tablet
W20260810 08:00:33.986238 15234 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.194:36265, user_credentials={real_user=slave}} blocked reactor thread for 68513.8us
I20260810 08:00:33.943413 14964 tablet_service.cc:1467] Processing CreateTablet for tablet e263a43f508643808cea1db210feffa4 (DEFAULT_TABLE table=default.Table_d34e1afecbcf418483dec42d2f76f8d9 [id=22b1e5a875e142ecb7fe1ba06f144e77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:33.993608 14964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e263a43f508643808cea1db210feffa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:33.998865 14689 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_d5c08115c8b24ad48a9686e48a45ab82" } modify_external_catalogs: true
I20260810 08:00:33.999439 14689 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_d5c08115c8b24ad48a9686e48a45ab82" } modify_external_catalogs: true
W20260810 08:00:34.002372 14692 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.14.57.194:55587 (request call id 50) took 894 ms (client timeout 1000 ms). Trace:
I20260810 08:00:34.008754 15122 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ac554bd37b045a2a8cf2743f4645f28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:29 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:34.009703 15479 tablet_replica.cc:330] T 3ac554bd37b045a2a8cf2743f4645f28 P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:34.010663 15479 raft_consensus.cc:2229] T 3ac554bd37b045a2a8cf2743f4645f28 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:34.011178 15479 raft_consensus.cc:2258] T 3ac554bd37b045a2a8cf2743f4645f28 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:34.013013 15479 ts_tablet_manager.cc:1905] T 3ac554bd37b045a2a8cf2743f4645f28 P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:34.016546 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "579bac622ddf40bf80d01d66ae2fa100" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:34.051281 15831 raft_consensus.cc:482] T b5a0e0af742340b2945be2102768b307 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:34.052070 15831 raft_consensus.cc:504] T b5a0e0af742340b2945be2102768b307 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:34.061292 14986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "389b30cba2b149e9ab0de8c191808db2" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:34.062923 15490 tablet_bootstrap.cc:654] T 4ca688b4031c41a9884829fbd0be1f33 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:34.067802 15437 tablet_bootstrap.cc:654] T 68bc39707f44457fbf57c217ad827348 P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
W20260810 08:00:34.025964 14686 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3855a2b4d90a47ad918c7a90cc8c7e4a consensus state has no leader
I20260810 08:00:34.062330 14688 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_02bc67e213c241fabead65c8dc785aa8" } modify_external_catalogs: true
I20260810 08:00:34.069357 14688 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_02bc67e213c241fabead65c8dc785aa8" } modify_external_catalogs: true
W20260810 08:00:34.075337 14692 rpcz_store.cc:269] 0810 08:00:33.107720 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:33.118836 (+ 11116us) service_pool.cc:227] Handling call
0810 08:00:34.002299 (+883463us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30830,"reported_tablets":11}
W20260810 08:00:34.080775 15857 log.cc:924] Time spent T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be: Append to log took a long time: real 0.052s user 0.001s sys 0.002s
I20260810 08:00:34.082018 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "389b30cba2b149e9ab0de8c191808db2" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
W20260810 08:00:34.092736 15076 leader_election.cc:341] T e13b91d61063446f9d1454da4f8bb706 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
W20260810 08:00:34.093221 15079 leader_election.cc:341] T 389b30cba2b149e9ab0de8c191808db2 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:34.093536 15076 leader_election.cc:302] T e13b91d61063446f9d1454da4f8bb706 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:34.094594 15146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a0e0af742340b2945be2102768b307" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:34.095634 15146 raft_consensus.cc:2454] T b5a0e0af742340b2945be2102768b307 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 0.
I20260810 08:00:34.095732 15795 raft_consensus.cc:2726] T e13b91d61063446f9d1454da4f8bb706 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 08:00:34.095983 15076 leader_election.cc:341] T 579bac622ddf40bf80d01d66ae2fa100 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:34.096632 15076 leader_election.cc:302] T 579bac622ddf40bf80d01d66ae2fa100 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:34.092928 15122 tablet_service.cc:1467] Processing CreateTablet for tablet e263a43f508643808cea1db210feffa4 (DEFAULT_TABLE table=default.Table_d34e1afecbcf418483dec42d2f76f8d9 [id=22b1e5a875e142ecb7fe1ba06f144e77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:34.099076 15122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e263a43f508643808cea1db210feffa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:34.102366 14922 leader_election.cc:302] T b5a0e0af742340b2945be2102768b307 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224; no voters:
I20260810 08:00:34.109935 15889 raft_consensus.cc:2726] T 579bac622ddf40bf80d01d66ae2fa100 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:34.120003 15857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 4 replicates
I20260810 08:00:34.126072 15118 tablet_service.cc:1467] Processing CreateTablet for tablet 8895d102ebcc4dff9e0e82f480824548 (DEFAULT_TABLE table=default.Table_62105ef471f3450bb3aabefb32baa96d [id=b9cf06fbbf5a4f4db8c7b7431e4c9f35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:34.128055 15118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8895d102ebcc4dff9e0e82f480824548. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:34.137854 15857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be: Bootstrap complete.
W20260810 08:00:34.149046 14694 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e263a43f508643808cea1db210feffa4 consensus state has no leader
I20260810 08:00:34.171099 15490 tablet_bootstrap.cc:492] T 4ca688b4031c41a9884829fbd0be1f33 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:34.171635 15490 ts_tablet_manager.cc:1397] T 4ca688b4031c41a9884829fbd0be1f33 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.307s user 0.006s sys 0.010s
I20260810 08:00:34.175647 15490 raft_consensus.cc:348] T 4ca688b4031c41a9884829fbd0be1f33 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:34.176317 15490 raft_consensus.cc:374] T 4ca688b4031c41a9884829fbd0be1f33 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:34.176554 15490 raft_consensus.cc:729] T 4ca688b4031c41a9884829fbd0be1f33 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:34.183271 15884 consensus_queue.cc:1035] T 2d295305b5a843eaa84c8ad666355b84 P 3fc5e568f3724f158b616c303efb9eae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 08:00:34.185904 15884 raft_consensus.cc:482] T 4381ccb8711744ff8f9766ef49c53e28 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 08:00:34.182934 15234 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.57.193:43601, user_credentials={real_user=slave}} blocked reactor thread for 101655us
I20260810 08:00:34.193358 15879 consensus_queue.cc:1035] T 25de7bf64bf6450ba17b7748568bc0ed P 3fc5e568f3724f158b616c303efb9eae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260810 08:00:34.196772 15234 leader_election.cc:341] T 68bc39707f44457fbf57c217ad827348 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:34.201201 15267 tablet_service.cc:1514] Processing DeleteTablet for tablet d3b41c4912f840c18913fe29c1af9fd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:33 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:34.204203 15267 tablet_service.cc:1467] Processing CreateTablet for tablet e263a43f508643808cea1db210feffa4 (DEFAULT_TABLE table=default.Table_d34e1afecbcf418483dec42d2f76f8d9 [id=22b1e5a875e142ecb7fe1ba06f144e77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:34.204783 15478 ts_tablet_manager.cc:1918] T 728f8f80b3bb4b10871d8b400f25377b P 3fc5e568f3724f158b616c303efb9eae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 08:00:34.198340 15439 tablet_bootstrap.cc:492] T 924c0f73638f48a88c08d45d459132e1 P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:34.211623 15478 log.cc:1196] T 728f8f80b3bb4b10871d8b400f25377b P 3fc5e568f3724f158b616c303efb9eae: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal/wals/728f8f80b3bb4b10871d8b400f25377b
I20260810 08:00:34.212478 15478 ts_tablet_manager.cc:1939] T 728f8f80b3bb4b10871d8b400f25377b P 3fc5e568f3724f158b616c303efb9eae: Deleting consensus metadata
I20260810 08:00:34.213377 15490 consensus_queue.cc:260] T 4ca688b4031c41a9884829fbd0be1f33 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:34.186790 15884 raft_consensus.cc:504] T 4381ccb8711744ff8f9766ef49c53e28 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:34.224573 15884 leader_election.cc:288] T 4381ccb8711744ff8f9766ef49c53e28 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:34.226490 15267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e263a43f508643808cea1db210feffa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:34.230417 15479 ts_tablet_manager.cc:1918] T 3ac554bd37b045a2a8cf2743f4645f28 P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:34.236420 15479 log.cc:1196] T 3ac554bd37b045a2a8cf2743f4645f28 P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/3ac554bd37b045a2a8cf2743f4645f28
I20260810 08:00:34.238210 15479 ts_tablet_manager.cc:1939] T 3ac554bd37b045a2a8cf2743f4645f28 P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:34.232368 15490 ts_tablet_manager.cc:1428] T 4ca688b4031c41a9884829fbd0be1f33 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.060s user 0.001s sys 0.006s
I20260810 08:00:34.243714 15490 tablet_bootstrap.cc:492] T e162a89dcf5a4890aa5a93dfeadd131e P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:34.247762 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 124) took 2780 ms. Trace:
I20260810 08:00:34.248235 15478 rpcz_store.cc:276] 0810 08:00:31.466943 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:31.476744 (+ 9801us) service_pool.cc:227] Handling call
0810 08:00:33.648591 (+2171847us) ts_tablet_manager.cc:1145] Deleting tablet 728f8f80b3bb4b10871d8b400f25377b
0810 08:00:33.648635 (+ 44us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:33.867472 (+218837us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:34.204518 (+337046us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:34.247688 (+ 43170us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2162086}
I20260810 08:00:34.249512 15478 tablet_replica.cc:330] T d0761b090d8147b8a5537b0ed51108a9 P 3fc5e568f3724f158b616c303efb9eae: stopping tablet replica
I20260810 08:00:34.251251 15478 raft_consensus.cc:2229] T d0761b090d8147b8a5537b0ed51108a9 P 3fc5e568f3724f158b616c303efb9eae [term 1 LEADER]: Raft consensus shutting down.
I20260810 08:00:34.251693 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet 3ac554bd37b045a2a8cf2743f4645f28 (table default.Table_e4e38aa2c28146cabb97071119b73368 [id=e356df0e1219419c9eaab96530f49ed7]) successfully deleted
I20260810 08:00:34.252736 15478 pending_rounds.cc:62] T d0761b090d8147b8a5537b0ed51108a9 P 3fc5e568f3724f158b616c303efb9eae: Trying to abort 1 pending ops.
I20260810 08:00:34.253105 15478 pending_rounds.cc:68] T d0761b090d8147b8a5537b0ed51108a9 P 3fc5e568f3724f158b616c303efb9eae: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316884819617275904 op_type: NO_OP noop_request { }
I20260810 08:00:34.253559 15478 raft_consensus.cc:2866] T d0761b090d8147b8a5537b0ed51108a9 P 3fc5e568f3724f158b616c303efb9eae [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260810 08:00:34.254050 15478 raft_consensus.cc:2258] T d0761b090d8147b8a5537b0ed51108a9 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:34.256652 15478 ts_tablet_manager.cc:1905] T d0761b090d8147b8a5537b0ed51108a9 P 3fc5e568f3724f158b616c303efb9eae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:34.263523 14694 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_8f57e64670f74bc3b08fa1e79718260b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:34.265169 14694 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8f57e64670f74bc3b08fa1e79718260b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:34.266376 14695 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_97e91b68f06d4cca9ba065f2e6a07b0f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:34.268136 14695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97e91b68f06d4cca9ba065f2e6a07b0f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:34.276038 14686 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_bfb2fd6d4f974132b165f34aedb323bc" } new_table_name: "default.Table_b825156514e8420c8592158108c8d065" modify_external_catalogs: true
I20260810 08:00:34.280409 15112 tablet_service.cc:1467] Processing CreateTablet for tablet d25928e4c39d45e69c3efeb36f03aece (DEFAULT_TABLE table=default.Table_93c814deac714a3288f3980340acbc65 [id=4c70468b9e8649849cd0e79d59be6bb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:34.281538 15112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d25928e4c39d45e69c3efeb36f03aece. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:34.303947 15490 tablet_bootstrap.cc:654] T e162a89dcf5a4890aa5a93dfeadd131e P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:34.306715 15478 ts_tablet_manager.cc:1918] T d0761b090d8147b8a5537b0ed51108a9 P 3fc5e568f3724f158b616c303efb9eae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 08:00:34.308046 14688 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet 579bac622ddf40bf80d01d66ae2fa100
I20260810 08:00:34.308565 14692 catalog_manager.cc:5587] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265) does not have the latest schema for tablet 389b30cba2b149e9ab0de8c191808db2 (table default.Table_e6b03850596042a6a30ee45ea0720b86 [id=6b0af0270f5d40e1a37b309b130be42a]). Expected version 1 got 0
I20260810 08:00:34.309670 14689 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet 2d295305b5a843eaa84c8ad666355b84
I20260810 08:00:34.310385 15478 log.cc:1196] T d0761b090d8147b8a5537b0ed51108a9 P 3fc5e568f3724f158b616c303efb9eae: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal/wals/d0761b090d8147b8a5537b0ed51108a9
I20260810 08:00:34.311297 15478 ts_tablet_manager.cc:1939] T d0761b090d8147b8a5537b0ed51108a9 P 3fc5e568f3724f158b616c303efb9eae: Deleting consensus metadata
I20260810 08:00:34.312728 15111 tablet_service.cc:1514] Processing DeleteTablet for tablet 579bac622ddf40bf80d01d66ae2fa100 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:34 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:34.314452 15111 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d295305b5a843eaa84c8ad666355b84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:33 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:34.326354 15280 tablet_service.cc:1467] Processing CreateTablet for tablet 8895d102ebcc4dff9e0e82f480824548 (DEFAULT_TABLE table=default.Table_62105ef471f3450bb3aabefb32baa96d [id=b9cf06fbbf5a4f4db8c7b7431e4c9f35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:34.329838 15280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8895d102ebcc4dff9e0e82f480824548. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:34.338304 15111 tablet_service.cc:1467] Processing CreateTablet for tablet 8c846b25718242ecbe0ac5dcc7a9c239 (DEFAULT_TABLE table=default.Table_73aba277eb504cc1917f096e97f4fd50 [id=2b9900d6a50549929028d26f1f1ccd4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:34.339393 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 125) took 2861 ms. Trace:
I20260810 08:00:34.340567 14689 catalog_manager.cc:5495] T 68bc39707f44457fbf57c217ad827348 P 3fc5e568f3724f158b616c303efb9eae reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fc5e568f3724f158b616c303efb9eae (127.14.57.195). New cstate: current_term: 1 leader_uuid: "3fc5e568f3724f158b616c303efb9eae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } health_report { overall_health: UNKNOWN } } }
I20260810 08:00:34.342103 14689 catalog_manager.cc:5587] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729) does not have the latest schema for tablet 68bc39707f44457fbf57c217ad827348 (table default.Table_4f3c632faa1e4c2d86acf10a656ee7c7 [id=fbd03ee96b004797a503841f4cc7249a]). Expected version 1 got 0
I20260810 08:00:34.326890 15275 tablet_service.cc:1467] Processing CreateTablet for tablet d25928e4c39d45e69c3efeb36f03aece (DEFAULT_TABLE table=default.Table_93c814deac714a3288f3980340acbc65 [id=4c70468b9e8649849cd0e79d59be6bb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:34.345703 15478 rpcz_store.cc:276] 0810 08:00:31.477830 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:31.481657 (+ 3827us) service_pool.cc:227] Handling call
0810 08:00:34.249329 (+2767672us) ts_tablet_manager.cc:1145] Deleting tablet d0761b090d8147b8a5537b0ed51108a9
0810 08:00:34.249385 (+ 56us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:34.276562 (+ 27177us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:34.306478 (+ 29916us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:34.339322 (+ 32844us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2549431}
I20260810 08:00:34.348381 15479 tablet_replica.cc:330] T 579bac622ddf40bf80d01d66ae2fa100 P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:34.349872 15479 raft_consensus.cc:2229] T 579bac622ddf40bf80d01d66ae2fa100 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:34.350405 15479 raft_consensus.cc:2258] T 579bac622ddf40bf80d01d66ae2fa100 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:34.352129 15479 ts_tablet_manager.cc:1905] T 579bac622ddf40bf80d01d66ae2fa100 P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:34.349115 15111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c846b25718242ecbe0ac5dcc7a9c239. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:34.352641 14688 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_d2ac9955e5b74f2791b2e54ed8b42e4c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:34.354070 14693 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_7ed048f56dd943beb001f4add1ee3332" } new_table_name: "default.Table_cf3389ce04894b849aa937e491c58c8f" modify_external_catalogs: true
I20260810 08:00:34.342967 15831 leader_election.cc:288] T b5a0e0af742340b2945be2102768b307 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:34.355516 15904 raft_consensus.cc:2781] T b5a0e0af742340b2945be2102768b307 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:34.355872 15904 raft_consensus.cc:482] T b5a0e0af742340b2945be2102768b307 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:34.356164 15904 raft_consensus.cc:3037] T b5a0e0af742340b2945be2102768b307 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:34.358850 15902 raft_consensus.cc:482] T 1f427f4093e5417a9ca79b1805b0675e P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:34.359278 15902 raft_consensus.cc:504] T 1f427f4093e5417a9ca79b1805b0675e P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
W20260810 08:00:34.359740 14688 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d2ac9955e5b74f2791b2e54ed8b42e4c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:34.361125 15902 leader_election.cc:288] T 1f427f4093e5417a9ca79b1805b0675e P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:34.366144 15478 tablet_replica.cc:330] T 4381ccb8711744ff8f9766ef49c53e28 P 3fc5e568f3724f158b616c303efb9eae: stopping tablet replica
I20260810 08:00:34.367002 15275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d25928e4c39d45e69c3efeb36f03aece. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:34.368961 15478 raft_consensus.cc:2229] T 4381ccb8711744ff8f9766ef49c53e28 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:34.370086 15478 raft_consensus.cc:2258] T 4381ccb8711744ff8f9766ef49c53e28 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:34.377859 15904 raft_consensus.cc:504] T b5a0e0af742340b2945be2102768b307 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:34.379136 15478 ts_tablet_manager.cc:1905] T 4381ccb8711744ff8f9766ef49c53e28 P 3fc5e568f3724f158b616c303efb9eae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:34.379938 15904 leader_election.cc:288] T b5a0e0af742340b2945be2102768b307 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Requested vote from peers 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:34.381560 15278 tablet_service.cc:1514] Processing DeleteTablet for tablet 579bac622ddf40bf80d01d66ae2fa100 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:34 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:34.385902 15278 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d295305b5a843eaa84c8ad666355b84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:33 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:34.387526 15278 tablet_service.cc:1467] Processing CreateTablet for tablet 8c846b25718242ecbe0ac5dcc7a9c239 (DEFAULT_TABLE table=default.Table_73aba277eb504cc1917f096e97f4fd50 [id=2b9900d6a50549929028d26f1f1ccd4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:34.388502 15278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c846b25718242ecbe0ac5dcc7a9c239. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:34.390403 15437 tablet_bootstrap.cc:492] T 68bc39707f44457fbf57c217ad827348 P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:34.396580 15437 ts_tablet_manager.cc:1397] T 68bc39707f44457fbf57c217ad827348 P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.535s user 0.008s sys 0.022s
I20260810 08:00:34.374605 15439 tablet_bootstrap.cc:654] T 924c0f73638f48a88c08d45d459132e1 P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:34.398886 15437 raft_consensus.cc:348] T 68bc39707f44457fbf57c217ad827348 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:34.399554 15437 raft_consensus.cc:374] T 68bc39707f44457fbf57c217ad827348 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:34.399807 15437 raft_consensus.cc:729] T 68bc39707f44457fbf57c217ad827348 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
W20260810 08:00:34.401959 15076 leader_election.cc:341] T 389b30cba2b149e9ab0de8c191808db2 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:34.402482 15076 leader_election.cc:302] T 389b30cba2b149e9ab0de8c191808db2 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:34.403610 15889 raft_consensus.cc:2726] T 389b30cba2b149e9ab0de8c191808db2 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:34.403949 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a0e0af742340b2945be2102768b307" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:34.407858 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f427f4093e5417a9ca79b1805b0675e" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:34.408702 14678 catalog_manager.cc:4841] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): tablet 728f8f80b3bb4b10871d8b400f25377b (table default.Table_4c0b6e5e875f43e088f4f0215cf39284 [id=b89eea0100b945c1851d323ce8c97591]) successfully deleted
I20260810 08:00:34.409680 14678 catalog_manager.cc:4841] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): tablet d0761b090d8147b8a5537b0ed51108a9 (table default.Table_f387e847280a4f048905e15c5003f58c [id=fc7978f30c9a49c8aef649a6d10d87ad]) successfully deleted
I20260810 08:00:34.410272 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a0e0af742340b2945be2102768b307" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae"
I20260810 08:00:34.412321 14986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8bb1c44a7b4d17bddeff8b2fe81b35" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224"
I20260810 08:00:34.412896 14986 raft_consensus.cc:3037] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:34.418490 15111 tablet_service.cc:1467] Processing CreateTablet for tablet d3b41c4912f840c18913fe29c1af9fd9 (DEFAULT_TABLE table=default.Table_36a773333abe4fcbaf61c6209c63f796 [id=2b430196a3fd48bba4f3a407c6112f3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:34.419438 14986 raft_consensus.cc:2454] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 1.
I20260810 08:00:34.419770 15111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3b41c4912f840c18913fe29c1af9fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:34.432559 15890 raft_consensus.cc:2229] T 2cca8a18778c49d0be01363c31ef180b P a91b27762182454baada71e91ceab224 [term 1 LEADER]: Raft consensus shutting down.
I20260810 08:00:34.435274 15479 ts_tablet_manager.cc:1918] T 579bac622ddf40bf80d01d66ae2fa100 P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:34.435773 15479 log.cc:1196] T 579bac622ddf40bf80d01d66ae2fa100 P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/579bac622ddf40bf80d01d66ae2fa100
I20260810 08:00:34.436519 15479 ts_tablet_manager.cc:1939] T 579bac622ddf40bf80d01d66ae2fa100 P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:34.436882 15122 tablet_service.cc:1467] Processing CreateTablet for tablet 2a680a71369647a287297f10a4b202c7 (DEFAULT_TABLE table=default.Table_a10d38e0a2114fb4a8a59a240cd5808f [id=abc319f2c0eb4a9db52881d546439a21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:34.437935 15122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a680a71369647a287297f10a4b202c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:34.442816 15890 raft_consensus.cc:2258] T 2cca8a18778c49d0be01363c31ef180b P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:34.450778 15478 ts_tablet_manager.cc:1918] T 4381ccb8711744ff8f9766ef49c53e28 P 3fc5e568f3724f158b616c303efb9eae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20260810 08:00:34.451772 14686 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1ce493c78a30404ca15e5b2568446bdf has no consensus state
I20260810 08:00:34.452656 15478 log.cc:1196] T 4381ccb8711744ff8f9766ef49c53e28 P 3fc5e568f3724f158b616c303efb9eae: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal/wals/4381ccb8711744ff8f9766ef49c53e28
I20260810 08:00:34.454233 15478 ts_tablet_manager.cc:1939] T 4381ccb8711744ff8f9766ef49c53e28 P 3fc5e568f3724f158b616c303efb9eae: Deleting consensus metadata
I20260810 08:00:34.457787 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 131) took 2186 ms. Trace:
I20260810 08:00:34.462030 15478 rpcz_store.cc:276] 0810 08:00:32.271050 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:32.280766 (+ 9716us) service_pool.cc:227] Handling call
0810 08:00:34.365976 (+2085210us) ts_tablet_manager.cc:1145] Deleting tablet 4381ccb8711744ff8f9766ef49c53e28
0810 08:00:34.366023 (+ 47us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:34.404877 (+ 38854us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:34.450514 (+ 45637us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:34.457720 (+ 7206us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2067717}
I20260810 08:00:34.465162 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 132) took 2193 ms. Trace:
W20260810 08:00:34.469673 14918 leader_election.cc:341] T b5a0e0af742340b2945be2102768b307 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
W20260810 08:00:34.470714 14918 leader_election.cc:341] T 1f427f4093e5417a9ca79b1805b0675e P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
W20260810 08:00:34.471378 14918 leader_election.cc:341] T b5a0e0af742340b2945be2102768b307 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:34.468272 15478 rpcz_store.cc:276] 0810 08:00:32.271285 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:32.305898 (+ 34613us) service_pool.cc:227] Handling call
0810 08:00:34.464829 (+2158931us) ts_tablet_manager.cc:1145] Deleting tablet adada21bc1bf4f19a6347d0935bd74bd
0810 08:00:34.464872 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:34.465114 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2158244}
I20260810 08:00:34.487258 15479 tablet_replica.cc:330] T 2d295305b5a843eaa84c8ad666355b84 P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:34.488189 15479 raft_consensus.cc:2229] T 2d295305b5a843eaa84c8ad666355b84 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:34.488711 15479 raft_consensus.cc:2258] T 2d295305b5a843eaa84c8ad666355b84 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:34.492063 14691 catalog_manager.cc:6040] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: default.Table_1567aaa1d2ab44028745e82c29232682 [id=6cf63d3e931a4f6487074494254805bb] alter complete (version 4)
I20260810 08:00:34.493163 15490 tablet_bootstrap.cc:492] T e162a89dcf5a4890aa5a93dfeadd131e P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:34.493602 15490 ts_tablet_manager.cc:1397] T e162a89dcf5a4890aa5a93dfeadd131e P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.250s user 0.007s sys 0.009s
I20260810 08:00:34.487327 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 133) took 1570 ms. Trace:
I20260810 08:00:34.495339 15478 rpcz_store.cc:276] 0810 08:00:32.917099 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:32.917269 (+ 170us) service_pool.cc:227] Handling call
0810 08:00:34.486965 (+1569696us) ts_tablet_manager.cc:1145] Deleting tablet 3281dd757ff44d9498ef70ad372eb150
0810 08:00:34.487011 (+ 46us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:34.487258 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1569123}
I20260810 08:00:34.496030 15490 raft_consensus.cc:348] T e162a89dcf5a4890aa5a93dfeadd131e P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:34.496775 15490 raft_consensus.cc:374] T e162a89dcf5a4890aa5a93dfeadd131e P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:34.497061 15490 raft_consensus.cc:729] T e162a89dcf5a4890aa5a93dfeadd131e P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:34.497993 15478 tablet_replica.cc:330] T 2cca8a18778c49d0be01363c31ef180b P 3fc5e568f3724f158b616c303efb9eae: stopping tablet replica
I20260810 08:00:34.499934 15478 raft_consensus.cc:2229] T 2cca8a18778c49d0be01363c31ef180b P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:34.503423 15889 raft_consensus.cc:482] T 3281dd757ff44d9498ef70ad372eb150 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:34.503894 15889 raft_consensus.cc:504] T 3281dd757ff44d9498ef70ad372eb150 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
W20260810 08:00:34.472944 14922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.194:36265, user_credentials={real_user=slave}} blocked reactor thread for 370632us
I20260810 08:00:34.505029 15490 consensus_queue.cc:260] T e162a89dcf5a4890aa5a93dfeadd131e P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:34.490895 15890 ts_tablet_manager.cc:1905] T 2cca8a18778c49d0be01363c31ef180b P a91b27762182454baada71e91ceab224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:34.511510 15144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a0e0af742340b2945be2102768b307" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07"
I20260810 08:00:34.512456 15146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f427f4093e5417a9ca79b1805b0675e" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:34.512775 15144 raft_consensus.cc:3037] T b5a0e0af742340b2945be2102768b307 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:34.513065 15146 raft_consensus.cc:2454] T 1f427f4093e5417a9ca79b1805b0675e P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 0.
I20260810 08:00:34.513208 15437 consensus_queue.cc:260] T 68bc39707f44457fbf57c217ad827348 P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:34.515280 15437 ts_tablet_manager.cc:1428] T 68bc39707f44457fbf57c217ad827348 P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.118s user 0.005s sys 0.000s
I20260810 08:00:34.515645 15889 leader_election.cc:288] T 3281dd757ff44d9498ef70ad372eb150 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:34.519665 15144 raft_consensus.cc:2454] T b5a0e0af742340b2945be2102768b307 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 1.
I20260810 08:00:34.526482 15118 tablet_service.cc:1514] Processing DeleteTablet for tablet 3281dd757ff44d9498ef70ad372eb150 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:31 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:34.528600 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3281dd757ff44d9498ef70ad372eb150" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
W20260810 08:00:34.530200 14693 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet acb4a99ef0c643908aef8aae87860eff has no consensus state
I20260810 08:00:34.530946 14960 tablet_service.cc:1467] Processing CreateTablet for tablet d25928e4c39d45e69c3efeb36f03aece (DEFAULT_TABLE table=default.Table_93c814deac714a3288f3980340acbc65 [id=4c70468b9e8649849cd0e79d59be6bb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:34.531941 14960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d25928e4c39d45e69c3efeb36f03aece. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:34.531819 14953 tablet_service.cc:1467] Processing CreateTablet for tablet 8c846b25718242ecbe0ac5dcc7a9c239 (DEFAULT_TABLE table=default.Table_73aba277eb504cc1917f096e97f4fd50 [id=2b9900d6a50549929028d26f1f1ccd4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:34.532855 14953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c846b25718242ecbe0ac5dcc7a9c239. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:34.534821 14967 tablet_service.cc:1514] Processing DeleteTablet for tablet 579bac622ddf40bf80d01d66ae2fa100 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:34 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:34.535202 15199 heartbeater.cc:347] Connected to a master server at 127.14.57.252:33819
I20260810 08:00:34.536401 14967 tablet_service.cc:1467] Processing CreateTablet for tablet d3b41c4912f840c18913fe29c1af9fd9 (DEFAULT_TABLE table=default.Table_36a773333abe4fcbaf61c6209c63f796 [id=2b430196a3fd48bba4f3a407c6112f3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:34.537474 14967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3b41c4912f840c18913fe29c1af9fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:34.537890 14961 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d295305b5a843eaa84c8ad666355b84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:33 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:34.536401 14964 tablet_service.cc:1467] Processing CreateTablet for tablet 8895d102ebcc4dff9e0e82f480824548 (DEFAULT_TABLE table=default.Table_62105ef471f3450bb3aabefb32baa96d [id=b9cf06fbbf5a4f4db8c7b7431e4c9f35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:34.539695 14964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8895d102ebcc4dff9e0e82f480824548. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 08:00:34.546530 15079 leader_election.cc:341] T 3281dd757ff44d9498ef70ad372eb150 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:34.548484 15490 ts_tablet_manager.cc:1428] T e162a89dcf5a4890aa5a93dfeadd131e P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.055s user 0.006s sys 0.000s
I20260810 08:00:34.549543 14693 master_service.cc:946] ReplaceTablet: received request to replace tablet 20a316e404414b288a315a5386208111 from {username='slave'} at 127.0.0.1:42300
I20260810 08:00:34.550499 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet 579bac622ddf40bf80d01d66ae2fa100 (table default.Table_02bc67e213c241fabead65c8dc785aa8 [id=650bc0c97e7842798d575199f1ea662c]) successfully deleted
I20260810 08:00:34.560150 15479 ts_tablet_manager.cc:1905] T 2d295305b5a843eaa84c8ad666355b84 P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 08:00:34.560988 15234 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.57.194:36265, user_credentials={real_user=slave}} blocked reactor thread for 150937us
W20260810 08:00:34.561595 15234 leader_election.cc:341] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Not found: Tablet not found: cf8bb1c44a7b4d17bddeff8b2fe81b35
I20260810 08:00:34.562387 15818 consensus_queue.cc:1035] T 25de7bf64bf6450ba17b7748568bc0ed P 3fc5e568f3724f158b616c303efb9eae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 08:00:34.564054 15234 leader_election.cc:341] T 4381ccb8711744ff8f9766ef49c53e28 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Not found: Tablet not found: 4381ccb8711744ff8f9766ef49c53e28
I20260810 08:00:34.565996 15273 tablet_service.cc:1514] Processing DeleteTablet for tablet cf8bb1c44a7b4d17bddeff8b2fe81b35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:30 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:34.566504 14678 catalog_manager.cc:4841] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): tablet 4381ccb8711744ff8f9766ef49c53e28 (table default.Table_4f3d374da9fd4fcaaff35b3e4c3bdbf7 [id=f07affbf494748a88874b8f1edd3f7b6]) successfully deleted
W20260810 08:00:34.567521 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet adada21bc1bf4f19a6347d0935bd74bd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet adada21bc1bf4f19a6347d0935bd74bd already in progress: creating tablet
W20260810 08:00:34.568143 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet 3281dd757ff44d9498ef70ad372eb150 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3281dd757ff44d9498ef70ad372eb150 already in progress: creating tablet
I20260810 08:00:34.570556 14686 master_service.cc:946] ReplaceTablet: received request to replace tablet 0a60001d76604d5085c843961d52213f from {username='slave'} at 127.0.0.1:42300
I20260810 08:00:34.570945 15267 tablet_service.cc:1467] Processing CreateTablet for tablet d3b41c4912f840c18913fe29c1af9fd9 (DEFAULT_TABLE table=default.Table_36a773333abe4fcbaf61c6209c63f796 [id=2b430196a3fd48bba4f3a407c6112f3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:34.572031 15267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3b41c4912f840c18913fe29c1af9fd9. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 08:00:34.573886 14918 consensus_peers.cc:487] T 561cbc6a4eb14d3584f88ea26a0fea7a P a91b27762182454baada71e91ceab224 -> Peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Couldn't send request to peer 3fc5e568f3724f158b616c303efb9eae. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260810 08:00:34.574373 15280 tablet_service.cc:1467] Processing CreateTablet for tablet 2a680a71369647a287297f10a4b202c7 (DEFAULT_TABLE table=default.Table_a10d38e0a2114fb4a8a59a240cd5808f [id=abc319f2c0eb4a9db52881d546439a21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:34.575264 15280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a680a71369647a287297f10a4b202c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:34.584043 15478 raft_consensus.cc:2258] T 2cca8a18778c49d0be01363c31ef180b P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:34.586539 15478 ts_tablet_manager.cc:1905] T 2cca8a18778c49d0be01363c31ef180b P 3fc5e568f3724f158b616c303efb9eae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:34.589303 14691 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_de9a90e3e5d841bd97301a0672436fdf" } modify_external_catalogs: true
I20260810 08:00:34.589891 14691 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_de9a90e3e5d841bd97301a0672436fdf" } modify_external_catalogs: true
I20260810 08:00:34.591585 14692 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_7893219bf6754edbba16135d2605adce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:34.593111 14692 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7893219bf6754edbba16135d2605adce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:34.598898 14694 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_4f0bc756581c45958dc46729a9430f79" } modify_external_catalogs: true
I20260810 08:00:34.599395 14694 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_4f0bc756581c45958dc46729a9430f79" } modify_external_catalogs: true
I20260810 08:00:34.611140 14689 catalog_manager.cc:5587] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265) does not have the latest schema for tablet 4ca688b4031c41a9884829fbd0be1f33 (table default.Table_f8637789a86d4e0da9392352f395f0c3 [id=da2bb7febfe9421380f61df1393ea3b0]). Expected version 1 got 0
I20260810 08:00:34.611459 15439 tablet_bootstrap.cc:492] T 924c0f73638f48a88c08d45d459132e1 P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:34.611980 15439 ts_tablet_manager.cc:1397] T 924c0f73638f48a88c08d45d459132e1 P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.694s user 0.042s sys 0.040s
I20260810 08:00:34.618506 15439 raft_consensus.cc:348] T 924c0f73638f48a88c08d45d459132e1 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:34.619387 15439 raft_consensus.cc:374] T 924c0f73638f48a88c08d45d459132e1 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:34.619704 15439 raft_consensus.cc:729] T 924c0f73638f48a88c08d45d459132e1 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:34.620711 14693 catalog_manager.cc:5587] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601) does not have the latest schema for tablet 68bc39707f44457fbf57c217ad827348 (table default.Table_4f3c632faa1e4c2d86acf10a656ee7c7 [id=fbd03ee96b004797a503841f4cc7249a]). Expected version 1 got 0
I20260810 08:00:34.620479 15439 consensus_queue.cc:260] T 924c0f73638f48a88c08d45d459132e1 P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:34.632681 14961 tablet_service.cc:1467] Processing CreateTablet for tablet 2a680a71369647a287297f10a4b202c7 (DEFAULT_TABLE table=default.Table_a10d38e0a2114fb4a8a59a240cd5808f [id=abc319f2c0eb4a9db52881d546439a21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:34.636206 14961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a680a71369647a287297f10a4b202c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:34.637054 15490 tablet_bootstrap.cc:492] T 8a0e1ad5653f46b4861b3e39629fbd2c P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:34.645961 15437 tablet_bootstrap.cc:492] T 8364045e3fa2441e8dc1a1d87f73361d P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:34.647675 15439 ts_tablet_manager.cc:1428] T 924c0f73638f48a88c08d45d459132e1 P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.035s user 0.005s sys 0.001s
I20260810 08:00:34.648121 15490 tablet_bootstrap.cc:654] T 8a0e1ad5653f46b4861b3e39629fbd2c P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:34.650203 15479 ts_tablet_manager.cc:1918] T 2d295305b5a843eaa84c8ad666355b84 P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 08:00:34.652526 15479 log.cc:1196] T 2d295305b5a843eaa84c8ad666355b84 P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/2d295305b5a843eaa84c8ad666355b84
I20260810 08:00:34.653537 15479 ts_tablet_manager.cc:1939] T 2d295305b5a843eaa84c8ad666355b84 P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:34.654155 14960 tablet_service.cc:1514] Processing DeleteTablet for tablet 924c0f73638f48a88c08d45d459132e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 3281dd757ff44d9498ef70ad372eb150) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:34.658382 15439 tablet_bootstrap.cc:492] T 1f427f4093e5417a9ca79b1805b0675e P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:34.662925 14693 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_f0a0a164132a4d46891036b0fee9839a" } new_table_name: "default.Table_e0d4352887e04b38939d453e07a37ef9" modify_external_catalogs: true
I20260810 08:00:34.666606 14922 leader_election.cc:302] T 1f427f4093e5417a9ca79b1805b0675e P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224; no voters: 3fc5e568f3724f158b616c303efb9eae
I20260810 08:00:34.667165 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3281dd757ff44d9498ef70ad372eb150" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:34.668766 14690 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_41b7b5b83e1443748d982b0d63c29acd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:34.669073 14695 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_4bed023fc139438b8753c1e1cf1dd469"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:34.670223 14690 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41b7b5b83e1443748d982b0d63c29acd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 08:00:34.670357 14695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4bed023fc139438b8753c1e1cf1dd469 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:34.673101 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet 2d295305b5a843eaa84c8ad666355b84 (table default.Table_d5c08115c8b24ad48a9686e48a45ab82 [id=bb6d5450013a45b1bb6a85bc9e4e6d9c]) successfully deleted
I20260810 08:00:34.673741 15907 raft_consensus.cc:2781] T 1f427f4093e5417a9ca79b1805b0675e P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:34.671725 15234 leader_election.cc:302] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224; no voters: 9d7219686f304f62a54acc7497cbbd07
I20260810 08:00:34.674515 15907 raft_consensus.cc:482] T 1f427f4093e5417a9ca79b1805b0675e P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:34.675866 15907 raft_consensus.cc:3037] T 1f427f4093e5417a9ca79b1805b0675e P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:34.677037 15818 raft_consensus.cc:2781] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Leader election won for term 1
I20260810 08:00:34.677843 15818 raft_consensus.cc:686] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 3fc5e568f3724f158b616c303efb9eae [term 1 LEADER]: Becoming Leader. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Running, Role: LEADER
I20260810 08:00:34.679272 15818 consensus_queue.cc:237] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 3fc5e568f3724f158b616c303efb9eae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:34.688271 15479 tablet_replica.cc:330] T 3281dd757ff44d9498ef70ad372eb150 P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:34.689219 15479 raft_consensus.cc:2229] T 3281dd757ff44d9498ef70ad372eb150 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:34.689838 15479 raft_consensus.cc:2258] T 3281dd757ff44d9498ef70ad372eb150 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:34.678395 15478 ts_tablet_manager.cc:1918] T 2cca8a18778c49d0be01363c31ef180b P 3fc5e568f3724f158b616c303efb9eae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 08:00:34.691481 15478 log.cc:1196] T 2cca8a18778c49d0be01363c31ef180b P 3fc5e568f3724f158b616c303efb9eae: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal/wals/2cca8a18778c49d0be01363c31ef180b
I20260810 08:00:34.692266 15478 ts_tablet_manager.cc:1939] T 2cca8a18778c49d0be01363c31ef180b P 3fc5e568f3724f158b616c303efb9eae: Deleting consensus metadata
I20260810 08:00:34.694105 15479 ts_tablet_manager.cc:1905] T 3281dd757ff44d9498ef70ad372eb150 P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:34.703142 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 135) took 1667 ms. Trace:
I20260810 08:00:34.703510 15478 rpcz_store.cc:276] 0810 08:00:33.035623 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:33.035793 (+ 170us) service_pool.cc:227] Handling call
0810 08:00:34.497836 (+1462043us) ts_tablet_manager.cc:1145] Deleting tablet 2cca8a18778c49d0be01363c31ef180b
0810 08:00:34.497881 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:34.633881 (+136000us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:34.678145 (+ 44264us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:34.703074 (+ 24929us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":79059,"tablet-delete.queue_time_us":1461475}
I20260810 08:00:34.704799 15478 tablet_replica.cc:330] T d22a778bfe784b2f8a7b4d2216909f10 P 3fc5e568f3724f158b616c303efb9eae: stopping tablet replica
W20260810 08:00:34.705013 15234 leader_election.cc:341] T 4381ccb8711744ff8f9766ef49c53e28 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Not found: Tablet not found: 4381ccb8711744ff8f9766ef49c53e28
I20260810 08:00:34.705545 15478 raft_consensus.cc:2229] T d22a778bfe784b2f8a7b4d2216909f10 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:34.722918 15478 raft_consensus.cc:2258] T d22a778bfe784b2f8a7b4d2216909f10 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:34.714331 15234 leader_election.cc:302] T 4381ccb8711744ff8f9766ef49c53e28 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae; no voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224
W20260810 08:00:34.726048 15234 raft_consensus.cc:2670] T 4381ccb8711744ff8f9766ef49c53e28 P 3fc5e568f3724f158b616c303efb9eae: unable to run election callback: Service unavailable: Thread pool token was shut down
W20260810 08:00:34.726006 14922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.194:36265, user_credentials={real_user=slave}} blocked reactor thread for 59493.1us
I20260810 08:00:34.726681 14922 leader_election.cc:302] T b5a0e0af742340b2945be2102768b307 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224; no voters: 3fc5e568f3724f158b616c303efb9eae
I20260810 08:00:34.727960 15908 raft_consensus.cc:2781] T b5a0e0af742340b2945be2102768b307 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 08:00:34.728574 15908 raft_consensus.cc:686] T b5a0e0af742340b2945be2102768b307 P a91b27762182454baada71e91ceab224 [term 1 LEADER]: Becoming Leader. State: Replica: a91b27762182454baada71e91ceab224, State: Running, Role: LEADER
I20260810 08:00:34.728582 14687 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_6099c55f1d194b5f90373405fc5f4977"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:34.730472 14687 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6099c55f1d194b5f90373405fc5f4977 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:34.731191 15439 tablet_bootstrap.cc:654] T 1f427f4093e5417a9ca79b1805b0675e P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:34.730733 15908 consensus_queue.cc:237] T b5a0e0af742340b2945be2102768b307 P a91b27762182454baada71e91ceab224 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:34.751363 15478 ts_tablet_manager.cc:1905] T d22a778bfe784b2f8a7b4d2216909f10 P 3fc5e568f3724f158b616c303efb9eae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:34.753655 14688 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_780441b622754f37a2cb73df2f19c3a3" } new_table_name: "default.Table_fcf1860f7d494dddadc19e82024de3c1" modify_external_catalogs: true
I20260810 08:00:34.766852 15354 heartbeater.cc:347] Connected to a master server at 127.14.57.252:33819
I20260810 08:00:34.767588 15354 heartbeater.cc:503] Master 127.14.57.252:33819 was elected leader, sending a full tablet report...
I20260810 08:00:34.768141 15122 tablet_service.cc:1514] Processing DeleteTablet for tablet e162a89dcf5a4890aa5a93dfeadd131e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:29 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:34.770974 14686 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet 0a60001d76604d5085c843961d52213f
I20260810 08:00:34.772820 14686 catalog_manager.cc:6565] ReplaceTablet: tablet 0a60001d76604d5085c843961d52213f deleted and replaced by tablet 175af9840d564708a5dbb5cffc94d3f7
W20260810 08:00:34.782848 15076 leader_election.cc:341] T 3281dd757ff44d9498ef70ad372eb150 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:34.783527 15076 leader_election.cc:302] T 3281dd757ff44d9498ef70ad372eb150 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
W20260810 08:00:34.784479 15076 raft_consensus.cc:2670] T 3281dd757ff44d9498ef70ad372eb150 P 9d7219686f304f62a54acc7497cbbd07: unable to run election callback: Service unavailable: Thread pool token was shut down
W20260810 08:00:34.786542 15234 consensus_peers.cc:487] T 2d295305b5a843eaa84c8ad666355b84 P 3fc5e568f3724f158b616c303efb9eae -> Peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Couldn't send request to peer 9d7219686f304f62a54acc7497cbbd07. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20260810 08:00:34.790366 14922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.194:36265, user_credentials={real_user=slave}} blocked reactor thread for 63755.1us
I20260810 08:00:34.790941 14694 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet 5f2e51d10e7142938e03b2ef687b0edd
I20260810 08:00:34.794402 15267 tablet_service.cc:1514] Processing DeleteTablet for tablet 924c0f73638f48a88c08d45d459132e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 3281dd757ff44d9498ef70ad372eb150) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:34.795877 15267 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a60001d76604d5085c843961d52213f with delete_type TABLET_DATA_DELETED (replaced by tablet 175af9840d564708a5dbb5cffc94d3f7) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:34.797367 15818 consensus_queue.cc:1035] T 68bc39707f44457fbf57c217ad827348 P 3fc5e568f3724f158b616c303efb9eae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 08:00:34.799671 15490 tablet_bootstrap.cc:492] T 8a0e1ad5653f46b4861b3e39629fbd2c P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:34.800110 15490 ts_tablet_manager.cc:1397] T 8a0e1ad5653f46b4861b3e39629fbd2c P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.163s user 0.010s sys 0.009s
I20260810 08:00:34.801186 15111 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f2e51d10e7142938e03b2ef687b0edd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:34 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:34.802785 14678 catalog_manager.cc:4841] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): tablet 2cca8a18778c49d0be01363c31ef180b (table default.Table_1e20f010e84746b0a487c82539f9ab8a [id=82b31a19f2724a95938f0ae5b4700d97]) successfully deleted
I20260810 08:00:34.812815 15267 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f2e51d10e7142938e03b2ef687b0edd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:34 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:34.812608 15490 raft_consensus.cc:348] T 8a0e1ad5653f46b4861b3e39629fbd2c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:34.813432 15490 raft_consensus.cc:374] T 8a0e1ad5653f46b4861b3e39629fbd2c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:34.829348 15490 raft_consensus.cc:729] T 8a0e1ad5653f46b4861b3e39629fbd2c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:34.834004 15490 consensus_queue.cc:260] T 8a0e1ad5653f46b4861b3e39629fbd2c P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:34.843107 15439 tablet_bootstrap.cc:492] T 1f427f4093e5417a9ca79b1805b0675e P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:34.853003 15479 ts_tablet_manager.cc:1918] T 3281dd757ff44d9498ef70ad372eb150 P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:34.857262 15439 ts_tablet_manager.cc:1397] T 1f427f4093e5417a9ca79b1805b0675e P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.199s user 0.007s sys 0.011s
I20260810 08:00:34.863404 14960 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f2e51d10e7142938e03b2ef687b0edd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:34 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:34.836710 15490 ts_tablet_manager.cc:1428] T 8a0e1ad5653f46b4861b3e39629fbd2c P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.036s user 0.005s sys 0.002s
I20260810 08:00:34.866514 15490 tablet_bootstrap.cc:492] T 5f2e51d10e7142938e03b2ef687b0edd P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:34.863740 15122 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a60001d76604d5085c843961d52213f with delete_type TABLET_DATA_DELETED (replaced by tablet 175af9840d564708a5dbb5cffc94d3f7) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:34.876483 15490 tablet_bootstrap.cc:654] T 5f2e51d10e7142938e03b2ef687b0edd P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:34.879663 14694 catalog_manager.cc:5495] T b5a0e0af742340b2945be2102768b307 P a91b27762182454baada71e91ceab224 reported cstate change: term changed from 0 to 1, leader changed from <none> to a91b27762182454baada71e91ceab224 (127.14.57.193). New cstate: current_term: 1 leader_uuid: "a91b27762182454baada71e91ceab224" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } health_report { overall_health: UNKNOWN } } }
I20260810 08:00:34.881455 14694 catalog_manager.cc:5587] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601) does not have the latest schema for tablet b5a0e0af742340b2945be2102768b307 (table default.Table_8642a50d2ed24720a1912ec9ea2e2939 [id=fc52022e20044cc3bc88ac143e088e69]). Expected version 3 got 0
I20260810 08:00:34.861058 15439 raft_consensus.cc:348] T 1f427f4093e5417a9ca79b1805b0675e P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:34.886391 15439 raft_consensus.cc:374] T 1f427f4093e5417a9ca79b1805b0675e P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:34.886853 15439 raft_consensus.cc:729] T 1f427f4093e5417a9ca79b1805b0675e P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:34.877022 15479 log.cc:1196] T 3281dd757ff44d9498ef70ad372eb150 P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/3281dd757ff44d9498ef70ad372eb150
I20260810 08:00:34.888907 15479 ts_tablet_manager.cc:1939] T 3281dd757ff44d9498ef70ad372eb150 P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:34.890231 15479 tablet_replica.cc:330] T e162a89dcf5a4890aa5a93dfeadd131e P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
W20260810 08:00:34.893214 14679 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 0687d323788642f798465ece973702bb (table default.Table_c65aa418eb904b57aab18f8b403e8c8e [id=0d60b763c4804818ae26fa693a4fd808], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 0687d323788642f798465ece973702bb consensus state has no leader
I20260810 08:00:34.892383 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet 3281dd757ff44d9498ef70ad372eb150 (table default.Table_409addd050ee45b981bddfacfab24c03 [id=76166bd7cb584067b7d2e305603a840b]) successfully deleted
I20260810 08:00:34.891232 15479 raft_consensus.cc:2229] T e162a89dcf5a4890aa5a93dfeadd131e P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:34.894982 15479 raft_consensus.cc:2258] T e162a89dcf5a4890aa5a93dfeadd131e P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:34.898803 15919 raft_consensus.cc:482] T 924c0f73638f48a88c08d45d459132e1 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:34.899515 15919 raft_consensus.cc:504] T 924c0f73638f48a88c08d45d459132e1 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:34.902191 15479 ts_tablet_manager.cc:1905] T e162a89dcf5a4890aa5a93dfeadd131e P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:34.903853 15919 leader_election.cc:288] T 924c0f73638f48a88c08d45d459132e1 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:34.904059 14691 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet 521d16ada2424d3aa41f838a511131fb
I20260810 08:00:34.888455 15439 consensus_queue.cc:260] T 1f427f4093e5417a9ca79b1805b0675e P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
W20260810 08:00:34.905890 15234 leader_election.cc:341] T 924c0f73638f48a88c08d45d459132e1 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Not found: Tablet not found: 924c0f73638f48a88c08d45d459132e1
I20260810 08:00:34.907076 15122 tablet_service.cc:1514] Processing DeleteTablet for tablet 521d16ada2424d3aa41f838a511131fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:34 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:34.908452 15267 tablet_service.cc:1514] Processing DeleteTablet for tablet 521d16ada2424d3aa41f838a511131fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:34 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:34.913367 15439 ts_tablet_manager.cc:1428] T 1f427f4093e5417a9ca79b1805b0675e P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.056s user 0.007s sys 0.000s
I20260810 08:00:34.917347 15478 ts_tablet_manager.cc:1918] T d22a778bfe784b2f8a7b4d2216909f10 P 3fc5e568f3724f158b616c303efb9eae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:34.928848 15478 log.cc:1196] T d22a778bfe784b2f8a7b4d2216909f10 P 3fc5e568f3724f158b616c303efb9eae: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal/wals/d22a778bfe784b2f8a7b4d2216909f10
I20260810 08:00:34.930106 15478 ts_tablet_manager.cc:1939] T d22a778bfe784b2f8a7b4d2216909f10 P 3fc5e568f3724f158b616c303efb9eae: Deleting consensus metadata
W20260810 08:00:34.956434 14688 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8980090fc4ff4de68bbb690f30c3f46e has no consensus state
I20260810 08:00:34.975999 15439 tablet_bootstrap.cc:492] T b5a0e0af742340b2945be2102768b307 P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
W20260810 08:00:34.980268 14693 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1f427f4093e5417a9ca79b1805b0675e consensus state has no leader
I20260810 08:00:34.981716 15889 raft_consensus.cc:482] T 4ca688b4031c41a9884829fbd0be1f33 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:34.982506 15889 raft_consensus.cc:504] T 4ca688b4031c41a9884829fbd0be1f33 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:34.989564 15889 leader_election.cc:288] T 4ca688b4031c41a9884829fbd0be1f33 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:34.994005 14967 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a60001d76604d5085c843961d52213f with delete_type TABLET_DATA_DELETED (replaced by tablet 175af9840d564708a5dbb5cffc94d3f7) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:34.912333 15857 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [term 1 FOLLOWER]: Replica starting. Triggering 4 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:35.003870 14693 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_c782f8d897974cef9ad4827497661a17" } modify_external_catalogs: true
I20260810 08:00:35.013979 14693 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_c782f8d897974cef9ad4827497661a17" } modify_external_catalogs: true
I20260810 08:00:35.032490 15889 raft_consensus.cc:482] T 8a0e1ad5653f46b4861b3e39629fbd2c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:35.033123 15889 raft_consensus.cc:504] T 8a0e1ad5653f46b4861b3e39629fbd2c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:35.040907 14694 catalog_manager.cc:5587] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265) does not have the latest schema for tablet 68bc39707f44457fbf57c217ad827348 (table default.Table_4f3c632faa1e4c2d86acf10a656ee7c7 [id=fbd03ee96b004797a503841f4cc7249a]). Expected version 1 got 0
I20260810 08:00:35.042481 15889 leader_election.cc:288] T 8a0e1ad5653f46b4861b3e39629fbd2c P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:35.045435 14953 tablet_service.cc:1514] Processing DeleteTablet for tablet 521d16ada2424d3aa41f838a511131fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:34 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:35.067162 15904 consensus_queue.cc:1035] T 561cbc6a4eb14d3584f88ea26a0fea7a P a91b27762182454baada71e91ceab224 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.031s
I20260810 08:00:35.084573 14988 raft_consensus.cc:3037] T 68bc39707f44457fbf57c217ad827348 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:35.099035 15122 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a0e1ad5653f46b4861b3e39629fbd2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:32 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:35.109360 14695 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_93c38e9ecbdd4eeeb498ad5ebb5eef2b" } modify_external_catalogs: true
I20260810 08:00:35.109999 14695 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_93c38e9ecbdd4eeeb498ad5ebb5eef2b" } modify_external_catalogs: true
I20260810 08:00:35.110895 14986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "924c0f73638f48a88c08d45d459132e1" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:35.111845 14986 raft_consensus.cc:2454] T 924c0f73638f48a88c08d45d459132e1 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 0.
I20260810 08:00:35.115994 15857 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e8cadb26fe0428a964f66ef8172f3be, State: Initialized, Role: FOLLOWER
I20260810 08:00:35.117429 15857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:35.128008 14986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ca688b4031c41a9884829fbd0be1f33" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:35.131521 15932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } } }
I20260810 08:00:35.132498 15932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [sys.catalog]: This master's current role is: FOLLOWER
I20260810 08:00:35.132233 15437 tablet_bootstrap.cc:654] T 8364045e3fa2441e8dc1a1d87f73361d P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:35.135847 14989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0e1ad5653f46b4861b3e39629fbd2c" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:35.142400 14692 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_7f4ac045f2d44f2e9aa549cb790acf5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:35.148267 15490 tablet_bootstrap.cc:492] T 5f2e51d10e7142938e03b2ef687b0edd P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:35.148245 15908 raft_consensus.cc:482] T 68bc39707f44457fbf57c217ad827348 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:35.148803 15908 raft_consensus.cc:504] T 68bc39707f44457fbf57c217ad827348 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:35.149446 15490 ts_tablet_manager.cc:1397] T 5f2e51d10e7142938e03b2ef687b0edd P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.283s user 0.008s sys 0.018s
I20260810 08:00:35.157130 14689 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_187379f079bf4283824c6b02b86271ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:35.157953 15079 leader_election.cc:341] T 4ca688b4031c41a9884829fbd0be1f33 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
W20260810 08:00:35.159303 15079 leader_election.cc:341] T 8a0e1ad5653f46b4861b3e39629fbd2c P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:35.164942 14688 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_043fe23b225140a7848446157e71c132" } new_table_name: "default.Table_c8ad27f2fffd4e729273033f0de8d50c" modify_external_catalogs: true
W20260810 08:00:35.167299 14692 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f4ac045f2d44f2e9aa549cb790acf5d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 08:00:35.171774 14689 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_187379f079bf4283824c6b02b86271ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:35.176160 15857 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [sys.catalog]: configured and running, proceeding with master startup.
I20260810 08:00:35.184885 14694 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_3aad4373b6464997880bb90d36fd1c6b" } new_table_name: "default.Table_4361cdd1d34f4d8fa5a656a1b06218b0" modify_external_catalogs: true
I20260810 08:00:35.175426 15490 raft_consensus.cc:348] T 5f2e51d10e7142938e03b2ef687b0edd P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:35.202558 15490 raft_consensus.cc:374] T 5f2e51d10e7142938e03b2ef687b0edd P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:35.203016 15490 raft_consensus.cc:729] T 5f2e51d10e7142938e03b2ef687b0edd P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:35.204654 15490 consensus_queue.cc:260] T 5f2e51d10e7142938e03b2ef687b0edd P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:35.215967 15479 ts_tablet_manager.cc:1918] T e162a89dcf5a4890aa5a93dfeadd131e P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:34.974706 15354 heartbeater.cc:503] Master 127.14.57.252:33819 was elected leader, sending a full tablet report...
W20260810 08:00:35.222007 14688 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fcd6d992683c4a5c886d7c5052da76b3 has no consensus state
I20260810 08:00:35.228188 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68bc39707f44457fbf57c217ad827348" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:35.221510 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 136) took 2184 ms. Trace:
I20260810 08:00:35.234840 15478 rpcz_store.cc:276] 0810 08:00:33.036780 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:33.036877 (+ 97us) service_pool.cc:227] Handling call
0810 08:00:34.704632 (+1667755us) ts_tablet_manager.cc:1145] Deleting tablet d22a778bfe784b2f8a7b4d2216909f10
0810 08:00:34.704677 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:34.862591 (+157914us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:34.917109 (+ 54518us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:35.221445 (+304336us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1667258}
I20260810 08:00:35.236199 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 137) took 2058 ms. Trace:
I20260810 08:00:35.236542 15478 rpcz_store.cc:276] 0810 08:00:33.177898 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:33.178008 (+ 110us) service_pool.cc:227] Handling call
0810 08:00:35.235796 (+2057788us) ts_tablet_manager.cc:1145] Deleting tablet 5990e7ea243843bb869de5dde3e6cd23
0810 08:00:35.235837 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:35.236132 (+ 295us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2057013}
I20260810 08:00:35.243136 14678 catalog_manager.cc:4841] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): tablet d22a778bfe784b2f8a7b4d2216909f10 (table default.Table_b7ee9352799b44f4b7bbf48355b3af11 [id=f008513c7e134e1992c6c8ad3e757bbb]) successfully deleted
I20260810 08:00:35.243322 15234 leader_election.cc:302] T 924c0f73638f48a88c08d45d459132e1 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224; no voters: 9d7219686f304f62a54acc7497cbbd07
W20260810 08:00:35.243775 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet 5990e7ea243843bb869de5dde3e6cd23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5990e7ea243843bb869de5dde3e6cd23 already in progress: creating tablet
I20260810 08:00:35.246330 15302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0e1ad5653f46b4861b3e39629fbd2c" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:35.251534 15884 consensus_queue.cc:1035] T 68bc39707f44457fbf57c217ad827348 P 3fc5e568f3724f158b616c303efb9eae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260810 08:00:35.249876 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 138) took 2008 ms. Trace:
W20260810 08:00:35.260448 15907 consensus_meta.cc:287] T 1f427f4093e5417a9ca79b1805b0675e P a91b27762182454baada71e91ceab224: Time spent flushing consensus metadata: real 0.583s user 0.002s sys 0.008s
I20260810 08:00:35.260264 15478 rpcz_store.cc:276] 0810 08:00:33.241607 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:33.241744 (+ 137us) service_pool.cc:227] Handling call
0810 08:00:35.237453 (+1995709us) ts_tablet_manager.cc:1145] Deleting tablet 8a0e1ad5653f46b4861b3e39629fbd2c
0810 08:00:35.237491 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:35.249806 (+ 12315us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1995121}
I20260810 08:00:35.260809 15907 raft_consensus.cc:504] T 1f427f4093e5417a9ca79b1805b0675e P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:35.262148 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 139) took 2020 ms. Trace:
I20260810 08:00:35.262516 15478 rpcz_store.cc:276] 0810 08:00:33.241920 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:33.242033 (+ 113us) service_pool.cc:227] Handling call
0810 08:00:35.261347 (+2019314us) ts_tablet_manager.cc:1145] Deleting tablet 68048e2768cf441d906e675254175c9c
0810 08:00:35.261392 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:35.262090 (+ 698us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2018739}
I20260810 08:00:35.251258 15818 raft_consensus.cc:2781] T 924c0f73638f48a88c08d45d459132e1 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:35.264489 15818 raft_consensus.cc:482] T 924c0f73638f48a88c08d45d459132e1 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:35.264999 15818 raft_consensus.cc:3037] T 924c0f73638f48a88c08d45d459132e1 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Advancing to term 1
W20260810 08:00:35.265499 14694 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8e6236e9f37146f6956e7a46eb3df3d5 has no consensus state
I20260810 08:00:35.257242 15439 tablet_bootstrap.cc:654] T b5a0e0af742340b2945be2102768b307 P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:35.249559 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ca688b4031c41a9884829fbd0be1f33" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:35.282968 14694 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_1dcf09dc44214dae8e1ed6319bdc825f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:35.263725 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 144) took 1469 ms. Trace:
I20260810 08:00:35.284879 15478 rpcz_store.cc:276] 0810 08:00:33.794717 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:33.794893 (+ 176us) service_pool.cc:227] Handling call
0810 08:00:35.263407 (+1468514us) ts_tablet_manager.cc:1145] Deleting tablet a92d01d83dbf4b6ba47464dcefa8b2ae
0810 08:00:35.263443 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:35.263674 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1441595}
W20260810 08:00:35.285461 14694 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1dcf09dc44214dae8e1ed6319bdc825f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:35.286099 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 143) took 1500 ms. Trace:
I20260810 08:00:35.286407 15478 rpcz_store.cc:276] 0810 08:00:33.786980 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:33.792775 (+ 5795us) service_pool.cc:227] Handling call
0810 08:00:35.285758 (+1492983us) ts_tablet_manager.cc:1145] Deleting tablet 2fbda1b3b5b4482ba6f03f3e275d94e7
0810 08:00:35.285798 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:35.286046 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1455584}
I20260810 08:00:35.287528 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 145) took 1435 ms. Trace:
I20260810 08:00:35.287807 15478 rpcz_store.cc:276] 0810 08:00:33.852539 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:33.858100 (+ 5561us) service_pool.cc:227] Handling call
0810 08:00:35.287222 (+1429122us) ts_tablet_manager.cc:1145] Deleting tablet 461de46896ee492b9f7cd1202208c19a
0810 08:00:35.287267 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:35.287479 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1428012}
I20260810 08:00:35.288928 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 146) took 1088 ms. Trace:
I20260810 08:00:35.294989 14689 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_70ae7b2430d94c3ab90e0d4efc349216"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:35.296543 14689 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70ae7b2430d94c3ab90e0d4efc349216 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 08:00:35.295393 14922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.57.194:36265, user_credentials={real_user=slave}} blocked reactor thread for 293448us
W20260810 08:00:35.309137 15234 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.57.194:36265, user_credentials={real_user=slave}} blocked reactor thread for 63556.7us
W20260810 08:00:35.320626 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet 8a0e1ad5653f46b4861b3e39629fbd2c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a0e1ad5653f46b4861b3e39629fbd2c already in progress: creating tablet
I20260810 08:00:35.323814 15890 ts_tablet_manager.cc:1918] T 2cca8a18778c49d0be01363c31ef180b P a91b27762182454baada71e91ceab224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20260810 08:00:35.321434 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet 68048e2768cf441d906e675254175c9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68048e2768cf441d906e675254175c9c already in progress: creating tablet
W20260810 08:00:35.328500 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet a92d01d83dbf4b6ba47464dcefa8b2ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a92d01d83dbf4b6ba47464dcefa8b2ae already in progress: creating tablet
I20260810 08:00:35.331559 15439 tablet_bootstrap.cc:492] T b5a0e0af742340b2945be2102768b307 P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:35.334664 15890 log.cc:1196] T 2cca8a18778c49d0be01363c31ef180b P a91b27762182454baada71e91ceab224: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal/wals/2cca8a18778c49d0be01363c31ef180b
I20260810 08:00:35.154124 15908 leader_election.cc:288] T 68bc39707f44457fbf57c217ad827348 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:35.323076 15907 leader_election.cc:288] T 1f427f4093e5417a9ca79b1805b0675e P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Requested vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:35.336429 15890 ts_tablet_manager.cc:1939] T 2cca8a18778c49d0be01363c31ef180b P a91b27762182454baada71e91ceab224: Deleting consensus metadata
I20260810 08:00:35.339965 15437 tablet_bootstrap.cc:492] T 8364045e3fa2441e8dc1a1d87f73361d P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:35.340523 15437 ts_tablet_manager.cc:1397] T 8364045e3fa2441e8dc1a1d87f73361d P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.823s user 0.014s sys 0.004s
I20260810 08:00:35.340451 15890 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34630 (request call id 144) took 1492 ms. Trace:
I20260810 08:00:35.341784 15890 rpcz_store.cc:276] 0810 08:00:33.848246 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:33.848401 (+ 155us) service_pool.cc:227] Handling call
0810 08:00:33.928070 (+ 79669us) ts_tablet_manager.cc:1145] Deleting tablet 2cca8a18778c49d0be01363c31ef180b
0810 08:00:33.928119 (+ 49us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:34.442103 (+513984us) spinlock_profiling.cc:243] Waited 358 ms on lock 0x7b54000b22d4. stack: 00007feaedce8033 00007feaedce7f30 00007feaed869534 00000000004db2b4 00000000004db249 00007feaf4bf4471 00007feaf4c77352 00007feaf508d26c 00007feaf5c9acfa 00007feaf5ccae10 00007feaf5cb1e0d 00007feaf5cb1db9 00007feaf5cb1d49 00007feaf5cb1d11 00007feaf5cb0f3d 0000000000580784
0810 08:00:34.833004 (+390901us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:35.323400 (+490396us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:35.340392 (+ 16992us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":822737664,"tablet-delete.queue_time_us":78520}
I20260810 08:00:35.343197 15439 ts_tablet_manager.cc:1397] T b5a0e0af742340b2945be2102768b307 P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.367s user 0.011s sys 0.008s
I20260810 08:00:35.344301 15437 raft_consensus.cc:348] T 8364045e3fa2441e8dc1a1d87f73361d P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:35.267789 15490 ts_tablet_manager.cc:1428] T 5f2e51d10e7142938e03b2ef687b0edd P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.116s user 0.006s sys 0.002s
I20260810 08:00:35.314528 15144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68bc39707f44457fbf57c217ad827348" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:35.345649 15479 log.cc:1196] T e162a89dcf5a4890aa5a93dfeadd131e P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/e162a89dcf5a4890aa5a93dfeadd131e
I20260810 08:00:35.346349 15479 ts_tablet_manager.cc:1939] T e162a89dcf5a4890aa5a93dfeadd131e P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:35.346503 15490 tablet_bootstrap.cc:492] T 0a60001d76604d5085c843961d52213f P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
W20260810 08:00:35.318522 15076 leader_election.cc:341] T 8a0e1ad5653f46b4861b3e39629fbd2c P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:35.347008 15076 leader_election.cc:302] T 8a0e1ad5653f46b4861b3e39629fbd2c P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:35.338964 15818 raft_consensus.cc:504] T 924c0f73638f48a88c08d45d459132e1 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:35.327322 15889 raft_consensus.cc:482] T 5f2e51d10e7142938e03b2ef687b0edd P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:35.289251 15478 rpcz_store.cc:276] 0810 08:00:34.200747 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:34.200893 (+ 146us) service_pool.cc:227] Handling call
0810 08:00:35.288547 (+1087654us) ts_tablet_manager.cc:1145] Deleting tablet d3b41c4912f840c18913fe29c1af9fd9
0810 08:00:35.288585 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:35.288854 (+ 269us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1086523}
I20260810 08:00:35.350250 15889 raft_consensus.cc:504] T 5f2e51d10e7142938e03b2ef687b0edd P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:35.351553 15478 tablet_replica.cc:330] T 2d295305b5a843eaa84c8ad666355b84 P 3fc5e568f3724f158b616c303efb9eae: stopping tablet replica
I20260810 08:00:35.352917 15478 raft_consensus.cc:2229] T 2d295305b5a843eaa84c8ad666355b84 P 3fc5e568f3724f158b616c303efb9eae [term 1 LEADER]: Raft consensus shutting down.
I20260810 08:00:35.353739 15478 pending_rounds.cc:62] T 2d295305b5a843eaa84c8ad666355b84 P 3fc5e568f3724f158b616c303efb9eae: Trying to abort 1 pending ops.
I20260810 08:00:35.367256 15884 raft_consensus.cc:482] T 1f427f4093e5417a9ca79b1805b0675e P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:35.367877 15884 raft_consensus.cc:504] T 1f427f4093e5417a9ca79b1805b0675e P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:35.370483 15884 leader_election.cc:288] T 1f427f4093e5417a9ca79b1805b0675e P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:35.364146 15478 pending_rounds.cc:68] T 2d295305b5a843eaa84c8ad666355b84 P 3fc5e568f3724f158b616c303efb9eae: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7316884827538821120 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "2d295305b5a843eaa84c8ad666355b84" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.Table_d5c08115c8b24ad48a9686e48a45ab82" dest_uuid: "3fc5e568f3724f158b616c303efb9eae" new_extra_config { } }
W20260810 08:00:35.372424 15478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40692: Aborted: Op aborted
I20260810 08:00:35.372267 15144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f427f4093e5417a9ca79b1805b0675e" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:35.373104 15144 raft_consensus.cc:2454] T 1f427f4093e5417a9ca79b1805b0675e P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 0.
I20260810 08:00:35.374468 15234 leader_election.cc:302] T 1f427f4093e5417a9ca79b1805b0675e P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, 9d7219686f304f62a54acc7497cbbd07; no voters:
I20260810 08:00:35.375236 15884 raft_consensus.cc:2781] T 1f427f4093e5417a9ca79b1805b0675e P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:35.375645 15884 raft_consensus.cc:482] T 1f427f4093e5417a9ca79b1805b0675e P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:35.376025 15884 raft_consensus.cc:3037] T 1f427f4093e5417a9ca79b1805b0675e P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:35.376129 15941 raft_consensus.cc:2726] T 8a0e1ad5653f46b4861b3e39629fbd2c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 08:00:35.339187 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet 2fbda1b3b5b4482ba6f03f3e275d94e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2fbda1b3b5b4482ba6f03f3e275d94e7 already in progress: creating tablet
W20260810 08:00:35.381927 15234 leader_election.cc:341] T 924c0f73638f48a88c08d45d459132e1 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Not found: Tablet not found: 924c0f73638f48a88c08d45d459132e1
I20260810 08:00:35.373435 15818 leader_election.cc:288] T 924c0f73638f48a88c08d45d459132e1 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Requested vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
W20260810 08:00:35.382555 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet 461de46896ee492b9f7cd1202208c19a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 461de46896ee492b9f7cd1202208c19a already in progress: creating tablet
I20260810 08:00:35.383129 15478 raft_consensus.cc:2258] T 2d295305b5a843eaa84c8ad666355b84 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 08:00:35.383253 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet d3b41c4912f840c18913fe29c1af9fd9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d3b41c4912f840c18913fe29c1af9fd9 already in progress: creating tablet
W20260810 08:00:35.383868 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet 579bac622ddf40bf80d01d66ae2fa100 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 579bac622ddf40bf80d01d66ae2fa100 already in progress: creating tablet
I20260810 08:00:35.385362 15478 ts_tablet_manager.cc:1905] T 2d295305b5a843eaa84c8ad666355b84 P 3fc5e568f3724f158b616c303efb9eae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:35.402570 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet e162a89dcf5a4890aa5a93dfeadd131e (table default.Table_ae0fee60b7704372bf9ac04a22df6876 [id=bdcacd958d574265adebc1fd1cb6c1fa]) successfully deleted
I20260810 08:00:35.403582 15800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [term 1 FOLLOWER]: Advancing to term 2
I20260810 08:00:35.400841 15889 leader_election.cc:288] T 5f2e51d10e7142938e03b2ef687b0edd P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:35.402062 15479 tablet_replica.cc:330] T 5f2e51d10e7142938e03b2ef687b0edd P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:35.416546 15479 raft_consensus.cc:2229] T 5f2e51d10e7142938e03b2ef687b0edd P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:35.417179 15479 raft_consensus.cc:2258] T 5f2e51d10e7142938e03b2ef687b0edd P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:35.417896 15267 tablet_service.cc:1467] Processing CreateTablet for tablet 0b04fb8571b24d39a387e90c4624d1e4 (DEFAULT_TABLE table=default.Table_8f57e64670f74bc3b08fa1e79718260b [id=75d527322bfc4b7e9bfc1d4d270946f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:35.419049 15267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b04fb8571b24d39a387e90c4624d1e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:35.420320 15479 ts_tablet_manager.cc:1905] T 5f2e51d10e7142938e03b2ef687b0edd P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:35.420878 14687 catalog_manager.cc:5587] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265) does not have the latest schema for tablet 68bc39707f44457fbf57c217ad827348 (table default.Table_4f3c632faa1e4c2d86acf10a656ee7c7 [id=fbd03ee96b004797a503841f4cc7249a]). Expected version 1 got 0
I20260810 08:00:35.426239 14688 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_d9fa6a06be6e42d1b504e0741c5462be" } new_table_name: "default.Table_a5c1e1f8311b4dc19e3d89b7baecd114" modify_external_catalogs: true
I20260810 08:00:35.426903 14692 catalog_manager.cc:5587] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729) does not have the latest schema for tablet 1f427f4093e5417a9ca79b1805b0675e (table default.Table_e0d4352887e04b38939d453e07a37ef9 [id=11ca1b81b7704d33820eef06f5a25310]). Expected version 1 got 0
W20260810 08:00:35.376878 15076 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.195:37729, user_credentials={real_user=slave}} blocked reactor thread for 58449.8us
W20260810 08:00:35.429632 15076 leader_election.cc:341] T 4ca688b4031c41a9884829fbd0be1f33 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:35.429940 15122 tablet_service.cc:1467] Processing CreateTablet for tablet 0b04fb8571b24d39a387e90c4624d1e4 (DEFAULT_TABLE table=default.Table_8f57e64670f74bc3b08fa1e79718260b [id=75d527322bfc4b7e9bfc1d4d270946f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:35.407956 15490 tablet_bootstrap.cc:654] T 0a60001d76604d5085c843961d52213f P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:35.431061 15122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b04fb8571b24d39a387e90c4624d1e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:35.432395 14690 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_1939fe86b63e4117be863f56bfe7131b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:35.433892 14690 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1939fe86b63e4117be863f56bfe7131b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:35.434382 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f2e51d10e7142938e03b2ef687b0edd" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:35.441664 15280 tablet_service.cc:1467] Processing CreateTablet for tablet 777bfc89ede045308dd67396a6f7f14d (DEFAULT_TABLE table=default.Table_812e1f2d005b4c1a8edc57a1c48b7fa0 [id=2f9a7edf6d114777b7444e8d6b03588b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:35.442674 15280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 777bfc89ede045308dd67396a6f7f14d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:35.447952 15076 leader_election.cc:302] T 4ca688b4031c41a9884829fbd0be1f33 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
W20260810 08:00:35.454900 15076 leader_election.cc:341] T 5f2e51d10e7142938e03b2ef687b0edd P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:35.459131 15439 raft_consensus.cc:348] T b5a0e0af742340b2945be2102768b307 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:35.459977 15439 raft_consensus.cc:374] T b5a0e0af742340b2945be2102768b307 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:35.459887 14692 catalog_manager.cc:5587] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729) does not have the latest schema for tablet 68bc39707f44457fbf57c217ad827348 (table default.Table_4f3c632faa1e4c2d86acf10a656ee7c7 [id=fbd03ee96b004797a503841f4cc7249a]). Expected version 1 got 0
I20260810 08:00:35.460345 15439 raft_consensus.cc:729] T b5a0e0af742340b2945be2102768b307 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:35.461166 15439 consensus_queue.cc:260] T b5a0e0af742340b2945be2102768b307 P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:35.409073 15884 raft_consensus.cc:504] T 1f427f4093e5417a9ca79b1805b0675e P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:35.455730 15941 raft_consensus.cc:2726] T 4ca688b4031c41a9884829fbd0be1f33 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:35.476524 15884 leader_election.cc:288] T 1f427f4093e5417a9ca79b1805b0675e P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Requested vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:35.477211 15144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f427f4093e5417a9ca79b1805b0675e" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07"
I20260810 08:00:35.477977 15144 raft_consensus.cc:3037] T 1f427f4093e5417a9ca79b1805b0675e P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:35.492187 15354 heartbeater.cc:503] Master 127.14.57.252:33819 was elected leader, sending a full tablet report...
I20260810 08:00:35.493659 14686 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_8642a50d2ed24720a1912ec9ea2e2939" } new_table_name: "default.Table_9ad8721c3e1e4c549faa983d21798cc2" modify_external_catalogs: true
I20260810 08:00:35.499109 15439 ts_tablet_manager.cc:1428] T b5a0e0af742340b2945be2102768b307 P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.150s user 0.005s sys 0.000s
I20260810 08:00:35.499531 14694 catalog_manager.cc:5587] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729) does not have the latest schema for tablet b5a0e0af742340b2945be2102768b307 (table default.Table_8642a50d2ed24720a1912ec9ea2e2939 [id=fc52022e20044cc3bc88ac143e088e69]). Expected version 3 got 0
I20260810 08:00:35.506044 15122 tablet_service.cc:1467] Processing CreateTablet for tablet 777bfc89ede045308dd67396a6f7f14d (DEFAULT_TABLE table=default.Table_812e1f2d005b4c1a8edc57a1c48b7fa0 [id=2f9a7edf6d114777b7444e8d6b03588b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:35.507301 15122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 777bfc89ede045308dd67396a6f7f14d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:35.508646 15278 tablet_service.cc:1467] Processing CreateTablet for tablet c2e8af0ba0e24a1981b36d892d189fdd (DEFAULT_TABLE table=default.Table_c4fc713ca0b54050b0475b8b746658f7 [id=69d89f0f5a4e4e8a8814ea01bca243ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:35.509792 15278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2e8af0ba0e24a1981b36d892d189fdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:35.509871 15144 raft_consensus.cc:2454] T 1f427f4093e5417a9ca79b1805b0675e P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 1.
I20260810 08:00:35.511420 15234 leader_election.cc:302] T 1f427f4093e5417a9ca79b1805b0675e P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, 9d7219686f304f62a54acc7497cbbd07; no voters:
I20260810 08:00:35.512393 15919 raft_consensus.cc:2781] T 1f427f4093e5417a9ca79b1805b0675e P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Leader election won for term 1
I20260810 08:00:35.513134 15919 raft_consensus.cc:686] T 1f427f4093e5417a9ca79b1805b0675e P 3fc5e568f3724f158b616c303efb9eae [term 1 LEADER]: Becoming Leader. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Running, Role: LEADER
I20260810 08:00:35.513867 15111 tablet_service.cc:1467] Processing CreateTablet for tablet c2e8af0ba0e24a1981b36d892d189fdd (DEFAULT_TABLE table=default.Table_c4fc713ca0b54050b0475b8b746658f7 [id=69d89f0f5a4e4e8a8814ea01bca243ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:35.515566 15111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2e8af0ba0e24a1981b36d892d189fdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:35.514899 15919 consensus_queue.cc:237] T 1f427f4093e5417a9ca79b1805b0675e P 3fc5e568f3724f158b616c303efb9eae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:35.523399 15302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f427f4093e5417a9ca79b1805b0675e" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae"
I20260810 08:00:35.535749 15490 tablet_bootstrap.cc:492] T 0a60001d76604d5085c843961d52213f P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:35.536302 15490 ts_tablet_manager.cc:1397] T 0a60001d76604d5085c843961d52213f P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.190s user 0.010s sys 0.002s
I20260810 08:00:35.538748 14687 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_ca56170c42664687b01194771c981e30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:35.539861 15490 raft_consensus.cc:348] T 0a60001d76604d5085c843961d52213f P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
W20260810 08:00:35.540298 14687 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca56170c42664687b01194771c981e30 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:35.541724 15490 raft_consensus.cc:374] T 0a60001d76604d5085c843961d52213f P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:35.557171 15490 raft_consensus.cc:729] T 0a60001d76604d5085c843961d52213f P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:35.558868 15439 tablet_bootstrap.cc:492] T cb37a3a87f8b48a68c19510f306073c2 P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:35.562803 15490 consensus_queue.cc:260] T 0a60001d76604d5085c843961d52213f P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:35.572389 15936 raft_consensus.cc:482] T b5a0e0af742340b2945be2102768b307 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:35.573179 15936 raft_consensus.cc:504] T b5a0e0af742340b2945be2102768b307 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:35.574131 15118 tablet_service.cc:1467] Processing CreateTablet for tablet 5c770c439e204656aea89f6d8ab17e78 (DEFAULT_TABLE table=default.Table_5f6628b89c6a42e38f9e8e1b57979e48 [id=eef8b519e8fb4b089c52febdd194add9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:35.577342 15936 leader_election.cc:288] T b5a0e0af742340b2945be2102768b307 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:35.581015 15118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c770c439e204656aea89f6d8ab17e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:35.548936 15146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f427f4093e5417a9ca79b1805b0675e" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07"
I20260810 08:00:35.586925 15146 raft_consensus.cc:2379] T 1f427f4093e5417a9ca79b1805b0675e P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a91b27762182454baada71e91ceab224 in current term 1: Already voted for candidate 3fc5e568f3724f158b616c303efb9eae in this term.
I20260810 08:00:35.608223 15112 tablet_service.cc:1467] Processing CreateTablet for tablet acb4a99ef0c643908aef8aae87860eff (DEFAULT_TABLE table=default.Table_cf3389ce04894b849aa937e491c58c8f [id=94b606e4ef6b4b4aa0baefda58461961]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:35.609467 15490 ts_tablet_manager.cc:1428] T 0a60001d76604d5085c843961d52213f P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.073s user 0.000s sys 0.007s
I20260810 08:00:35.609573 15112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acb4a99ef0c643908aef8aae87860eff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:35.594764 15944 tablet.cc:1722] T 68bc39707f44457fbf57c217ad827348 P 3fc5e568f3724f158b616c303efb9eae: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 08:00:35.642318 15490 tablet_bootstrap.cc:492] T 68048e2768cf441d906e675254175c9c P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:35.653540 14693 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet abc607ab01f74b2c9c693d6dea73090c
I20260810 08:00:35.656566 15437 raft_consensus.cc:374] T 8364045e3fa2441e8dc1a1d87f73361d P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:35.656975 15437 raft_consensus.cc:729] T 8364045e3fa2441e8dc1a1d87f73361d P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:35.657781 15437 consensus_queue.cc:260] T 8364045e3fa2441e8dc1a1d87f73361d P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:35.658378 15113 tablet_service.cc:1514] Processing DeleteTablet for tablet abc607ab01f74b2c9c693d6dea73090c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:35 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:35.545471 14922 leader_election.cc:302] T 68bc39707f44457fbf57c217ad827348 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a91b27762182454baada71e91ceab224; no voters: 3fc5e568f3724f158b616c303efb9eae, 9d7219686f304f62a54acc7497cbbd07
I20260810 08:00:35.590024 14984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f427f4093e5417a9ca79b1805b0675e" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224"
I20260810 08:00:35.660724 14984 raft_consensus.cc:2379] T 1f427f4093e5417a9ca79b1805b0675e P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fc5e568f3724f158b616c303efb9eae in current term 1: Already voted for candidate a91b27762182454baada71e91ceab224 in this term.
I20260810 08:00:35.599149 14986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f427f4093e5417a9ca79b1805b0675e" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:35.662057 14986 raft_consensus.cc:2379] T 1f427f4093e5417a9ca79b1805b0675e P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fc5e568f3724f158b616c303efb9eae in current term 1: Already voted for candidate a91b27762182454baada71e91ceab224 in this term.
I20260810 08:00:35.606535 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f2e51d10e7142938e03b2ef687b0edd" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:35.628811 14989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "924c0f73638f48a88c08d45d459132e1" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224"
I20260810 08:00:35.663182 14989 raft_consensus.cc:3037] T 924c0f73638f48a88c08d45d459132e1 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:35.665711 14695 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet fc0169c909454088b90ae8a5e01c1ec6
I20260810 08:00:35.666819 15439 tablet_bootstrap.cc:654] T cb37a3a87f8b48a68c19510f306073c2 P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:35.670094 15113 tablet_service.cc:1514] Processing DeleteTablet for tablet fc0169c909454088b90ae8a5e01c1ec6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:35 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:35.673022 15479 ts_tablet_manager.cc:1918] T 5f2e51d10e7142938e03b2ef687b0edd P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:35.673496 15479 log.cc:1196] T 5f2e51d10e7142938e03b2ef687b0edd P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/5f2e51d10e7142938e03b2ef687b0edd
I20260810 08:00:35.673631 14989 raft_consensus.cc:2454] T 924c0f73638f48a88c08d45d459132e1 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 1.
I20260810 08:00:35.679194 14695 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_24b98b64df0645a1ac616bc044689f83"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:35.681102 14695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_24b98b64df0645a1ac616bc044689f83 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 08:00:35.675585 15079 leader_election.cc:341] T 5f2e51d10e7142938e03b2ef687b0edd P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:35.691295 15079 leader_election.cc:302] T 5f2e51d10e7142938e03b2ef687b0edd P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
W20260810 08:00:35.691946 15079 raft_consensus.cc:2670] T 5f2e51d10e7142938e03b2ef687b0edd P 9d7219686f304f62a54acc7497cbbd07: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260810 08:00:35.693893 15302 raft_consensus.cc:3037] T b5a0e0af742340b2945be2102768b307 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:35.679641 15890 tablet_replica.cc:330] T d0761b090d8147b8a5537b0ed51108a9 P a91b27762182454baada71e91ceab224: stopping tablet replica
I20260810 08:00:35.696089 15890 raft_consensus.cc:2229] T d0761b090d8147b8a5537b0ed51108a9 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:35.696623 15890 raft_consensus.cc:2258] T d0761b090d8147b8a5537b0ed51108a9 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:35.699069 15890 ts_tablet_manager.cc:1905] T d0761b090d8147b8a5537b0ed51108a9 P a91b27762182454baada71e91ceab224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:35.699350 15113 tablet_service.cc:1467] Processing CreateTablet for tablet 44a0c3158c5c489392ecbee546828da3 (DEFAULT_TABLE table=default.Table_d2ac9955e5b74f2791b2e54ed8b42e4c [id=b439386701844638b950b5f289173a08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:35.700546 15113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44a0c3158c5c489392ecbee546828da3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:35.704864 15439 tablet_bootstrap.cc:492] T cb37a3a87f8b48a68c19510f306073c2 P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:35.705296 15439 ts_tablet_manager.cc:1397] T cb37a3a87f8b48a68c19510f306073c2 P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.147s user 0.003s sys 0.010s
I20260810 08:00:35.707593 15439 raft_consensus.cc:348] T cb37a3a87f8b48a68c19510f306073c2 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:35.708256 15439 raft_consensus.cc:374] T cb37a3a87f8b48a68c19510f306073c2 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:35.708554 15439 raft_consensus.cc:729] T cb37a3a87f8b48a68c19510f306073c2 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:35.715268 15302 raft_consensus.cc:1261] T b5a0e0af742340b2945be2102768b307 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Refusing update from remote peer a91b27762182454baada71e91ceab224: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 08:00:35.719769 15904 raft_consensus.cc:2726] T 68bc39707f44457fbf57c217ad827348 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 08:00:35.722343 15479 ts_tablet_manager.cc:1939] T 5f2e51d10e7142938e03b2ef687b0edd P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
W20260810 08:00:35.724367 15234 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.57.193:43601, user_credentials={real_user=slave}} blocked reactor thread for 197709us
I20260810 08:00:35.724797 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet 5f2e51d10e7142938e03b2ef687b0edd (table default.Table_4f0bc756581c45958dc46729a9430f79 [id=87b8720fcdf143299d050c9546dcd91d]) successfully deleted
I20260810 08:00:35.727103 15278 tablet_service.cc:1467] Processing CreateTablet for tablet 5c770c439e204656aea89f6d8ab17e78 (DEFAULT_TABLE table=default.Table_5f6628b89c6a42e38f9e8e1b57979e48 [id=eef8b519e8fb4b089c52febdd194add9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:35.727272 15275 tablet_service.cc:1467] Processing CreateTablet for tablet acb4a99ef0c643908aef8aae87860eff (DEFAULT_TABLE table=default.Table_cf3389ce04894b849aa937e491c58c8f [id=94b606e4ef6b4b4aa0baefda58461961]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:35.728457 15275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acb4a99ef0c643908aef8aae87860eff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:35.728848 15904 consensus_queue.cc:1035] T b5a0e0af742340b2945be2102768b307 P a91b27762182454baada71e91ceab224 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 08:00:35.729173 15273 tablet_service.cc:1467] Processing CreateTablet for tablet 44a0c3158c5c489392ecbee546828da3 (DEFAULT_TABLE table=default.Table_d2ac9955e5b74f2791b2e54ed8b42e4c [id=b439386701844638b950b5f289173a08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:35.730460 15273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44a0c3158c5c489392ecbee546828da3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:35.731666 15144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a0e0af742340b2945be2102768b307" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:35.732471 15144 raft_consensus.cc:2379] T b5a0e0af742340b2945be2102768b307 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fc5e568f3724f158b616c303efb9eae in current term 1: Already voted for candidate a91b27762182454baada71e91ceab224 in this term.
I20260810 08:00:35.734308 15278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c770c439e204656aea89f6d8ab17e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:35.741940 15268 tablet_service.cc:1514] Processing DeleteTablet for tablet abc607ab01f74b2c9c693d6dea73090c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:35 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:35.742201 15271 tablet_service.cc:1514] Processing DeleteTablet for tablet fc0169c909454088b90ae8a5e01c1ec6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:35 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:35.743739 15271 tablet_service.cc:1467] Processing CreateTablet for tablet 876f18afb76b4cbeab544f7eae35808d (DEFAULT_TABLE table=default.Table_42d526ee0aeb4b6b8b7e99b05d6b67d2 [id=bc751f0653a742958190fc3fae5b6def]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:35.744879 15271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 876f18afb76b4cbeab544f7eae35808d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:35.744978 15119 tablet_service.cc:1467] Processing CreateTablet for tablet 876f18afb76b4cbeab544f7eae35808d (DEFAULT_TABLE table=default.Table_42d526ee0aeb4b6b8b7e99b05d6b67d2 [id=bc751f0653a742958190fc3fae5b6def]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:35.746511 14985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a0e0af742340b2945be2102768b307" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:35.748612 15116 tablet_service.cc:1467] Processing CreateTablet for tablet 8e6236e9f37146f6956e7a46eb3df3d5 (DEFAULT_TABLE table=default.Table_4361cdd1d34f4d8fa5a656a1b06218b0 [id=6bc3360c711f4209975c7f731892dfbb]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 08:00:35.750804 14961 consensus_meta.cc:287] T 2a680a71369647a287297f10a4b202c7 P a91b27762182454baada71e91ceab224: Time spent flushing consensus metadata: real 0.682s user 0.010s sys 0.007s
I20260810 08:00:35.752213 15490 tablet_bootstrap.cc:654] T 68048e2768cf441d906e675254175c9c P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
W20260810 08:00:35.753806 14688 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8b5a629a7f8041179a006fa412fb6b48 consensus state has no leader
I20260810 08:00:35.760712 14689 catalog_manager.cc:5587] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601) does not have the latest schema for tablet 68bc39707f44457fbf57c217ad827348 (table default.Table_4f3c632faa1e4c2d86acf10a656ee7c7 [id=fbd03ee96b004797a503841f4cc7249a]). Expected version 1 got 0
I20260810 08:00:35.762823 14690 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_d797c9d1654e4f828ed4b8a7cce2460d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:35.763232 14922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.194:36265, user_credentials={real_user=slave}} blocked reactor thread for 219118us
I20260810 08:00:35.764307 14922 leader_election.cc:302] T 1f427f4093e5417a9ca79b1805b0675e P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a91b27762182454baada71e91ceab224; no voters: 3fc5e568f3724f158b616c303efb9eae, 9d7219686f304f62a54acc7497cbbd07
W20260810 08:00:35.764528 14690 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d797c9d1654e4f828ed4b8a7cce2460d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:35.766912 14953 tablet_service.cc:1467] Processing CreateTablet for tablet 0b04fb8571b24d39a387e90c4624d1e4 (DEFAULT_TABLE table=default.Table_8f57e64670f74bc3b08fa1e79718260b [id=75d527322bfc4b7e9bfc1d4d270946f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:35.768306 14953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b04fb8571b24d39a387e90c4624d1e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:35.769411 15234 leader_election.cc:302] T 924c0f73638f48a88c08d45d459132e1 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224; no voters: 9d7219686f304f62a54acc7497cbbd07
I20260810 08:00:35.770210 15884 raft_consensus.cc:2781] T 924c0f73638f48a88c08d45d459132e1 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Leader election won for term 1
I20260810 08:00:35.770689 15884 raft_consensus.cc:686] T 924c0f73638f48a88c08d45d459132e1 P 3fc5e568f3724f158b616c303efb9eae [term 1 LEADER]: Becoming Leader. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Running, Role: LEADER
I20260810 08:00:35.771493 15884 consensus_queue.cc:237] T 924c0f73638f48a88c08d45d459132e1 P 3fc5e568f3724f158b616c303efb9eae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
W20260810 08:00:35.773444 15234 consensus_peers.cc:487] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 3fc5e568f3724f158b616c303efb9eae -> Peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Couldn't send request to peer 9d7219686f304f62a54acc7497cbbd07. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf8bb1c44a7b4d17bddeff8b2fe81b35. This is attempt 1: this message will repeat every 5th retry.
I20260810 08:00:35.774209 15234 leader_election.cc:302] T b5a0e0af742340b2945be2102768b307 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae; no voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224
I20260810 08:00:35.780710 15437 ts_tablet_manager.cc:1428] T 8364045e3fa2441e8dc1a1d87f73361d P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.440s user 0.004s sys 0.012s
I20260810 08:00:35.791021 15928 raft_consensus.cc:2726] T 1f427f4093e5417a9ca79b1805b0675e P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 08:00:35.791791 15928 raft_consensus.cc:482] T 8364045e3fa2441e8dc1a1d87f73361d P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:35.792217 15928 raft_consensus.cc:504] T 8364045e3fa2441e8dc1a1d87f73361d P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:35.709110 15439 consensus_queue.cc:260] T cb37a3a87f8b48a68c19510f306073c2 P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:35.794287 15928 leader_election.cc:288] T 8364045e3fa2441e8dc1a1d87f73361d P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:35.795667 15439 ts_tablet_manager.cc:1428] T cb37a3a87f8b48a68c19510f306073c2 P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.090s user 0.007s sys 0.000s
I20260810 08:00:35.796380 15947 raft_consensus.cc:2726] T b5a0e0af742340b2945be2102768b307 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:35.802637 14961 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34630 (request call id 156) took 1250 ms. Trace:
I20260810 08:00:35.806445 15933 mvcc.cc:205] Tried to move back new op lower bound from 7316884830931734528 to 7316884827074424832. Current Snapshot: MvccSnapshot[applied={T|T < 7316884830931734528}]
I20260810 08:00:35.824513 14961 rpcz_store.cc:276] 0810 08:00:34.553390 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:34.631169 (+ 77779us) service_pool.cc:227] Handling call
0810 08:00:34.635453 (+ 4284us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 08:00:34.635659 (+ 206us) ts_tablet_manager.cc:676] Creating new metadata...
0810 08:00:34.887180 (+251521us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:35.801965 (+914785us) tablet_replica.cc:205] Creating consensus instance
0810 08:00:35.802543 (+ 578us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 08:00:35.827831 15268 tablet_service.cc:1467] Processing CreateTablet for tablet 8980090fc4ff4de68bbb690f30c3f46e (DEFAULT_TABLE table=default.Table_fcf1860f7d494dddadc19e82024de3c1 [id=7835a5751ffe4cac8863cb9aec1e3a87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:35.826563 15437 tablet_bootstrap.cc:492] T a244ad3bda5d45f9b89f68b734338d21 P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:35.833665 15273 tablet_service.cc:1467] Processing CreateTablet for tablet 8e6236e9f37146f6956e7a46eb3df3d5 (DEFAULT_TABLE table=default.Table_4361cdd1d34f4d8fa5a656a1b06218b0 [id=6bc3360c711f4209975c7f731892dfbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:35.836267 15960 tablet.cc:1722] T 68bc39707f44457fbf57c217ad827348 P 9d7219686f304f62a54acc7497cbbd07: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 08:00:35.839573 14964 tablet_service.cc:1467] Processing CreateTablet for tablet c2e8af0ba0e24a1981b36d892d189fdd (DEFAULT_TABLE table=default.Table_c4fc713ca0b54050b0475b8b746658f7 [id=69d89f0f5a4e4e8a8814ea01bca243ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:35.840579 14964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2e8af0ba0e24a1981b36d892d189fdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:35.846325 15119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 876f18afb76b4cbeab544f7eae35808d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:35.854446 14961 tablet_service.cc:1467] Processing CreateTablet for tablet 777bfc89ede045308dd67396a6f7f14d (DEFAULT_TABLE table=default.Table_812e1f2d005b4c1a8edc57a1c48b7fa0 [id=2f9a7edf6d114777b7444e8d6b03588b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:35.855074 15112 tablet_service.cc:1467] Processing CreateTablet for tablet 8980090fc4ff4de68bbb690f30c3f46e (DEFAULT_TABLE table=default.Table_fcf1860f7d494dddadc19e82024de3c1 [id=7835a5751ffe4cac8863cb9aec1e3a87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:35.855783 14961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 777bfc89ede045308dd67396a6f7f14d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:35.857906 15112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8980090fc4ff4de68bbb690f30c3f46e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:35.859560 15884 consensus_queue.cc:1035] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 3fc5e568f3724f158b616c303efb9eae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 08:00:35.860118 15270 tablet_service.cc:1467] Processing CreateTablet for tablet fcd6d992683c4a5c886d7c5052da76b3 (DEFAULT_TABLE table=default.Table_c8ad27f2fffd4e729273033f0de8d50c [id=f171faf09b8742d9a10a58d37c96b943]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:35.832839 15968 tablet.cc:1722] T 68bc39707f44457fbf57c217ad827348 P a91b27762182454baada71e91ceab224: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 08:00:35.866115 14694 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_80fd70a445294836a1e5485e58cecde8" } modify_external_catalogs: true
I20260810 08:00:35.866709 14694 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_80fd70a445294836a1e5485e58cecde8" } modify_external_catalogs: true
I20260810 08:00:35.866684 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8364045e3fa2441e8dc1a1d87f73361d" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:35.884428 15944 tablet.cc:1722] T b5a0e0af742340b2945be2102768b307 P 3fc5e568f3724f158b616c303efb9eae: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260810 08:00:35.886646 15478 ts_tablet_manager.cc:1918] T 2d295305b5a843eaa84c8ad666355b84 P 3fc5e568f3724f158b616c303efb9eae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 08:00:35.887038 15478 log.cc:1196] T 2d295305b5a843eaa84c8ad666355b84 P 3fc5e568f3724f158b616c303efb9eae: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal/wals/2d295305b5a843eaa84c8ad666355b84
I20260810 08:00:35.887728 15478 ts_tablet_manager.cc:1939] T 2d295305b5a843eaa84c8ad666355b84 P 3fc5e568f3724f158b616c303efb9eae: Deleting consensus metadata
I20260810 08:00:35.890911 14686 catalog_manager.cc:5495] T 1f427f4093e5417a9ca79b1805b0675e P 3fc5e568f3724f158b616c303efb9eae reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fc5e568f3724f158b616c303efb9eae (127.14.57.195). New cstate: current_term: 1 leader_uuid: "3fc5e568f3724f158b616c303efb9eae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } health_report { overall_health: UNKNOWN } } }
I20260810 08:00:35.892374 14686 catalog_manager.cc:5587] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729) does not have the latest schema for tablet 1f427f4093e5417a9ca79b1805b0675e (table default.Table_e0d4352887e04b38939d453e07a37ef9 [id=11ca1b81b7704d33820eef06f5a25310]). Expected version 1 got 0
I20260810 08:00:35.893393 14686 catalog_manager.cc:5587] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729) does not have the latest schema for tablet b5a0e0af742340b2945be2102768b307 (table default.Table_9ad8721c3e1e4c549faa983d21798cc2 [id=fc52022e20044cc3bc88ac143e088e69]). Expected version 4 got 0
I20260810 08:00:35.910923 15965 tablet.cc:1722] T b5a0e0af742340b2945be2102768b307 P a91b27762182454baada71e91ceab224: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260810 08:00:35.913872 15268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8980090fc4ff4de68bbb690f30c3f46e. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 08:00:35.918116 14678 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 8b5a629a7f8041179a006fa412fb6b48 (table default.Table_a5c1e1f8311b4dc19e3d89b7baecd114 [id=b333dde37a3e456c9f5b361e5125c8cc], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8b5a629a7f8041179a006fa412fb6b48 consensus state has no leader
I20260810 08:00:35.925122 14967 tablet_service.cc:1467] Processing CreateTablet for tablet acb4a99ef0c643908aef8aae87860eff (DEFAULT_TABLE table=default.Table_cf3389ce04894b849aa937e491c58c8f [id=94b606e4ef6b4b4aa0baefda58461961]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:35.925380 14960 tablet_service.cc:1467] Processing CreateTablet for tablet 44a0c3158c5c489392ecbee546828da3 (DEFAULT_TABLE table=default.Table_d2ac9955e5b74f2791b2e54ed8b42e4c [id=b439386701844638b950b5f289173a08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:35.927457 15270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcd6d992683c4a5c886d7c5052da76b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:35.928615 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 152) took 1546 ms. Trace:
I20260810 08:00:35.928956 15478 rpcz_store.cc:276] 0810 08:00:34.382428 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:34.385715 (+ 3287us) service_pool.cc:227] Handling call
0810 08:00:35.351351 (+965636us) ts_tablet_manager.cc:1145] Deleting tablet 2d295305b5a843eaa84c8ad666355b84
0810 08:00:35.351406 (+ 55us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:35.507473 (+156067us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:35.886425 (+378952us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:35.928557 (+ 42132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":965058}
I20260810 08:00:35.930060 15478 tablet_replica.cc:330] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 3fc5e568f3724f158b616c303efb9eae: stopping tablet replica
I20260810 08:00:35.930722 15478 raft_consensus.cc:2229] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 3fc5e568f3724f158b616c303efb9eae [term 1 LEADER]: Raft consensus shutting down.
I20260810 08:00:35.931356 15478 pending_rounds.cc:62] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 3fc5e568f3724f158b616c303efb9eae: Trying to abort 1 pending ops.
I20260810 08:00:35.931555 15478 pending_rounds.cc:68] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 3fc5e568f3724f158b616c303efb9eae: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316884826890682368 op_type: NO_OP noop_request { }
I20260810 08:00:35.931917 15478 raft_consensus.cc:2866] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 3fc5e568f3724f158b616c303efb9eae [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260810 08:00:35.932227 15478 raft_consensus.cc:2258] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:35.932575 14689 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_b825156514e8420c8592158108c8d065" } modify_external_catalogs: true
I20260810 08:00:35.933418 14689 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_b825156514e8420c8592158108c8d065" } modify_external_catalogs: true
I20260810 08:00:35.933992 15273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e6236e9f37146f6956e7a46eb3df3d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:35.934273 15439 tablet_bootstrap.cc:492] T 561cbc6a4eb14d3584f88ea26a0fea7a P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:35.872304 15490 tablet_bootstrap.cc:492] T 68048e2768cf441d906e675254175c9c P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:35.896139 15479 tablet_replica.cc:330] T 0a60001d76604d5085c843961d52213f P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:35.921838 15111 tablet_service.cc:1467] Processing CreateTablet for tablet 1ce493c78a30404ca15e5b2568446bdf (DEFAULT_TABLE table=default.Table_b825156514e8420c8592158108c8d065 [id=7bab6eb9b0c041a28751b184f70e132a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:35.923020 15113 tablet_service.cc:1467] Processing CreateTablet for tablet fcd6d992683c4a5c886d7c5052da76b3 (DEFAULT_TABLE table=default.Table_c8ad27f2fffd4e729273033f0de8d50c [id=f171faf09b8742d9a10a58d37c96b943]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:35.943017 14678 catalog_manager.cc:4841] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): tablet 2d295305b5a843eaa84c8ad666355b84 (table default.Table_d5c08115c8b24ad48a9686e48a45ab82 [id=bb6d5450013a45b1bb6a85bc9e4e6d9c]) successfully deleted
I20260810 08:00:35.943387 15280 tablet_service.cc:1467] Processing CreateTablet for tablet 1ce493c78a30404ca15e5b2568446bdf (DEFAULT_TABLE table=default.Table_b825156514e8420c8592158108c8d065 [id=7bab6eb9b0c041a28751b184f70e132a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:35.944476 15280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ce493c78a30404ca15e5b2568446bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:35.944746 15479 raft_consensus.cc:2229] T 0a60001d76604d5085c843961d52213f P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:35.945322 15479 raft_consensus.cc:2258] T 0a60001d76604d5085c843961d52213f P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:35.946710 15116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e6236e9f37146f6956e7a46eb3df3d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:35.947325 15439 tablet_bootstrap.cc:654] T 561cbc6a4eb14d3584f88ea26a0fea7a P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:35.950345 15111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ce493c78a30404ca15e5b2568446bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:35.951700 15437 tablet_bootstrap.cc:654] T a244ad3bda5d45f9b89f68b734338d21 P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:35.952203 15490 ts_tablet_manager.cc:1397] T 68048e2768cf441d906e675254175c9c P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.310s user 0.008s sys 0.004s
I20260810 08:00:35.954967 15490 raft_consensus.cc:348] T 68048e2768cf441d906e675254175c9c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:35.955881 15490 raft_consensus.cc:374] T 68048e2768cf441d906e675254175c9c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:35.953481 14967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acb4a99ef0c643908aef8aae87860eff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:35.956172 15490 raft_consensus.cc:729] T 68048e2768cf441d906e675254175c9c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:35.956737 14964 tablet_service.cc:1467] Processing CreateTablet for tablet 5c770c439e204656aea89f6d8ab17e78 (DEFAULT_TABLE table=default.Table_5f6628b89c6a42e38f9e8e1b57979e48 [id=eef8b519e8fb4b089c52febdd194add9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:35.957443 15113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcd6d992683c4a5c886d7c5052da76b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:35.961971 14964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c770c439e204656aea89f6d8ab17e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:35.955401 14965 tablet_service.cc:1514] Processing DeleteTablet for tablet fc0169c909454088b90ae8a5e01c1ec6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:35 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:35.963496 14687 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_82ef137d777f463c843b60a4dc498741" } new_table_name: "default.Table_1ed39e9362eb43ca8b58ae159fdd59d8" modify_external_catalogs: true
I20260810 08:00:35.963652 14688 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_65d76ebe6be54a089add93b91dafcb9e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:35.965133 14688 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65d76ebe6be54a089add93b91dafcb9e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:35.971499 15479 ts_tablet_manager.cc:1905] T 0a60001d76604d5085c843961d52213f P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:35.972890 15478 ts_tablet_manager.cc:1905] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 3fc5e568f3724f158b616c303efb9eae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:35.974222 14960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44a0c3158c5c489392ecbee546828da3. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 08:00:35.953379 14918 leader_election.cc:341] T 8364045e3fa2441e8dc1a1d87f73361d P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
W20260810 08:00:35.977833 14918 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.195:37729, user_credentials={real_user=slave}} blocked reactor thread for 70858.5us
I20260810 08:00:35.955025 14951 tablet_service.cc:1514] Processing DeleteTablet for tablet abc607ab01f74b2c9c693d6dea73090c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:35 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:35.981393 15490 consensus_queue.cc:260] T 68048e2768cf441d906e675254175c9c P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:35.987274 15490 ts_tablet_manager.cc:1428] T 68048e2768cf441d906e675254175c9c P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.035s user 0.007s sys 0.000s
I20260810 08:00:35.988358 14952 tablet_service.cc:1467] Processing CreateTablet for tablet 8e6236e9f37146f6956e7a46eb3df3d5 (DEFAULT_TABLE table=default.Table_4361cdd1d34f4d8fa5a656a1b06218b0 [id=6bc3360c711f4209975c7f731892dfbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:35.990159 14681 catalog_manager.cc:4841] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): tablet 2cca8a18778c49d0be01363c31ef180b (table default.Table_1e20f010e84746b0a487c82539f9ab8a [id=82b31a19f2724a95938f0ae5b4700d97]) successfully deleted
I20260810 08:00:35.994598 14957 tablet_service.cc:1467] Processing CreateTablet for tablet 8980090fc4ff4de68bbb690f30c3f46e (DEFAULT_TABLE table=default.Table_fcf1860f7d494dddadc19e82024de3c1 [id=7835a5751ffe4cac8863cb9aec1e3a87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:35.995172 14963 tablet_service.cc:1467] Processing CreateTablet for tablet 1ce493c78a30404ca15e5b2568446bdf (DEFAULT_TABLE table=default.Table_b825156514e8420c8592158108c8d065 [id=7bab6eb9b0c041a28751b184f70e132a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:35.993525 14955 tablet_service.cc:1467] Processing CreateTablet for tablet fcd6d992683c4a5c886d7c5052da76b3 (DEFAULT_TABLE table=default.Table_c8ad27f2fffd4e729273033f0de8d50c [id=f171faf09b8742d9a10a58d37c96b943]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:35.954595 14954 tablet_service.cc:1467] Processing CreateTablet for tablet 876f18afb76b4cbeab544f7eae35808d (DEFAULT_TABLE table=default.Table_42d526ee0aeb4b6b8b7e99b05d6b67d2 [id=bc751f0653a742958190fc3fae5b6def]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:35.996873 14951 tablet_service.cc:1514] Processing DeleteTablet for tablet 8364045e3fa2441e8dc1a1d87f73361d with delete_type TABLET_DATA_DELETED (replaced by tablet fd97cbfb16bf4ee2b4eed1e8bbbc7224) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:36.007830 14692 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_8aecb3c6298849c38bcafee965161494" } new_table_name: "default.Table_437480e7bb1145b59dc24585f0639e62" modify_external_catalogs: true
I20260810 08:00:36.012445 15278 tablet_service.cc:1467] Processing CreateTablet for tablet 6173d97ad84c43b3848c1b673d7a35b7 (DEFAULT_TABLE table=default.Table_d48d3754e2e84c28b58869ee05f4ea95 [id=0ae329e15cfe4dc0a3e0de17663b058d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:36.013589 15278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6173d97ad84c43b3848c1b673d7a35b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:36.031049 15490 tablet_bootstrap.cc:492] T 2fbda1b3b5b4482ba6f03f3e275d94e7 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:36.052732 15479 ts_tablet_manager.cc:1918] T 0a60001d76604d5085c843961d52213f P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:36.062069 15479 log.cc:1196] T 0a60001d76604d5085c843961d52213f P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/0a60001d76604d5085c843961d52213f
I20260810 08:00:36.064394 15479 ts_tablet_manager.cc:1939] T 0a60001d76604d5085c843961d52213f P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:36.094317 15478 ts_tablet_manager.cc:1918] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 3fc5e568f3724f158b616c303efb9eae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 08:00:36.094792 15478 log.cc:1196] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 3fc5e568f3724f158b616c303efb9eae: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal/wals/cf8bb1c44a7b4d17bddeff8b2fe81b35
I20260810 08:00:36.095510 15478 ts_tablet_manager.cc:1939] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P 3fc5e568f3724f158b616c303efb9eae: Deleting consensus metadata
I20260810 08:00:36.107074 14951 tablet_service.cc:1467] Processing CreateTablet for tablet 6173d97ad84c43b3848c1b673d7a35b7 (DEFAULT_TABLE table=default.Table_d48d3754e2e84c28b58869ee05f4ea95 [id=0ae329e15cfe4dc0a3e0de17663b058d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:36.085572 14952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e6236e9f37146f6956e7a46eb3df3d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:36.111699 14694 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet ed8f3e52959b4a0c8ed2a053e05e5bd6
I20260810 08:00:36.118166 15122 tablet_service.cc:1467] Processing CreateTablet for tablet 6173d97ad84c43b3848c1b673d7a35b7 (DEFAULT_TABLE table=default.Table_d48d3754e2e84c28b58869ee05f4ea95 [id=0ae329e15cfe4dc0a3e0de17663b058d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:36.123433 14690 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_68ea7153b43948c19bb9a37d1cddc777"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:36.123872 15110 tablet_service.cc:1467] Processing CreateTablet for tablet 0493eb96a2a042718527ddcd9524d799 (DEFAULT_TABLE table=default.Table_9a8eb842ebf244d5abbfb0f43373ca08 [id=23114ccea26d48079f910570c9580a71]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 08:00:36.124946 14690 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68ea7153b43948c19bb9a37d1cddc777 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:36.128464 15144 raft_consensus.cc:1261] T b5a0e0af742340b2945be2102768b307 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Refusing update from remote peer a91b27762182454baada71e91ceab224: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 08:00:36.131549 15890 ts_tablet_manager.cc:1918] T d0761b090d8147b8a5537b0ed51108a9 P a91b27762182454baada71e91ceab224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:36.132229 15890 log.cc:1196] T d0761b090d8147b8a5537b0ed51108a9 P a91b27762182454baada71e91ceab224: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal/wals/d0761b090d8147b8a5537b0ed51108a9
I20260810 08:00:36.138128 14957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8980090fc4ff4de68bbb690f30c3f46e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:36.141664 15479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43002 (request call id 148) took 1361 ms. Trace:
I20260810 08:00:36.141916 15122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6173d97ad84c43b3848c1b673d7a35b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:36.141989 15479 rpcz_store.cc:276] 0810 08:00:34.779882 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:34.863472 (+ 83590us) service_pool.cc:227] Handling call
0810 08:00:35.813476 (+950004us) ts_tablet_manager.cc:1145] Deleting tablet 0a60001d76604d5085c843961d52213f
0810 08:00:35.896000 (+ 82524us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:36.040396 (+144396us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:36.052443 (+ 12047us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:36.141585 (+ 89142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":939563}
I20260810 08:00:36.143239 15479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43002 (request call id 150) took 1236 ms. Trace:
I20260810 08:00:36.143517 15479 rpcz_store.cc:276] 0810 08:00:34.906634 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:34.906793 (+ 159us) service_pool.cc:227] Handling call
0810 08:00:36.142934 (+1236141us) ts_tablet_manager.cc:1145] Deleting tablet 521d16ada2424d3aa41f838a511131fb
0810 08:00:36.142973 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:36.143188 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1235356}
I20260810 08:00:36.144335 15479 tablet_replica.cc:330] T 8a0e1ad5653f46b4861b3e39629fbd2c P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:36.119625 15941 raft_consensus.cc:482] T 68048e2768cf441d906e675254175c9c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:36.145704 14954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 876f18afb76b4cbeab544f7eae35808d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:36.146189 15941 raft_consensus.cc:504] T 68048e2768cf441d906e675254175c9c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:36.149729 14963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ce493c78a30404ca15e5b2568446bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:36.120955 15118 tablet_service.cc:1514] Processing DeleteTablet for tablet 68048e2768cf441d906e675254175c9c with delete_type TABLET_DATA_DELETED (replaced by tablet f221dc79f1104fd0ab225c09da0df457) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:36.154050 15110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0493eb96a2a042718527ddcd9524d799. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:36.154111 14955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcd6d992683c4a5c886d7c5052da76b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:36.163257 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet 0a60001d76604d5085c843961d52213f (table default.Table_814fa7cd026940c385ff0c1e0ec55a01 [id=f2c914866c3f418caf5e08a2ad03a91b]) successfully deleted
W20260810 08:00:36.164038 14681 catalog_manager.cc:4821] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): delete failed for tablet 521d16ada2424d3aa41f838a511131fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 521d16ada2424d3aa41f838a511131fb already in progress: creating tablet
I20260810 08:00:36.165377 15118 tablet_service.cc:1514] Processing DeleteTablet for tablet ed8f3e52959b4a0c8ed2a053e05e5bd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:35 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:36.172878 14989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68048e2768cf441d906e675254175c9c" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:36.177069 15479 raft_consensus.cc:2229] T 8a0e1ad5653f46b4861b3e39629fbd2c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:36.178745 15479 raft_consensus.cc:2258] T 8a0e1ad5653f46b4861b3e39629fbd2c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260810 08:00:36.179764 15079 leader_election.cc:341] T 68048e2768cf441d906e675254175c9c P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:36.180667 15479 ts_tablet_manager.cc:1905] T 8a0e1ad5653f46b4861b3e39629fbd2c P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:36.184410 14694 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_85256d2eb76e488ea952541bfdf13953"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:36.185954 14694 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_85256d2eb76e488ea952541bfdf13953 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:36.191015 14951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6173d97ad84c43b3848c1b673d7a35b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:36.199391 14693 catalog_manager.cc:5587] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601) does not have the latest schema for tablet 68bc39707f44457fbf57c217ad827348 (table default.Table_4f3c632faa1e4c2d86acf10a656ee7c7 [id=fbd03ee96b004797a503841f4cc7249a]). Expected version 1 got 0
I20260810 08:00:36.212034 15118 tablet_service.cc:1467] Processing CreateTablet for tablet f221dc79f1104fd0ab225c09da0df457 (DEFAULT_TABLE table=default.Table_4ff94124f5cb4710a0ad3ff1781efcd3 [id=0b1956596de94b569ebd9f1816e4b699]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:36.226799 15890 ts_tablet_manager.cc:1939] T d0761b090d8147b8a5537b0ed51108a9 P a91b27762182454baada71e91ceab224: Deleting consensus metadata
I20260810 08:00:36.197435 15941 leader_election.cc:288] T 68048e2768cf441d906e675254175c9c P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
W20260810 08:00:36.221210 14922 leader_election.cc:341] T 8364045e3fa2441e8dc1a1d87f73361d P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Not found: Tablet not found: 8364045e3fa2441e8dc1a1d87f73361d
I20260810 08:00:36.235167 14922 leader_election.cc:302] T 8364045e3fa2441e8dc1a1d87f73361d P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a91b27762182454baada71e91ceab224; no voters: 3fc5e568f3724f158b616c303efb9eae, 9d7219686f304f62a54acc7497cbbd07
I20260810 08:00:36.235682 15941 raft_consensus.cc:482] T a244ad3bda5d45f9b89f68b734338d21 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:36.237059 15490 tablet_bootstrap.cc:654] T 2fbda1b3b5b4482ba6f03f3e275d94e7 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:36.236354 15941 raft_consensus.cc:504] T a244ad3bda5d45f9b89f68b734338d21 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:36.243235 15951 raft_consensus.cc:2726] T 8364045e3fa2441e8dc1a1d87f73361d P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:36.244148 14689 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet 1ce493c78a30404ca15e5b2568446bdf
I20260810 08:00:36.263590 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 157) took 1697 ms. Trace:
I20260810 08:00:36.263950 15478 rpcz_store.cc:276] 0810 08:00:34.565729 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:34.565836 (+ 107us) service_pool.cc:227] Handling call
0810 08:00:35.929926 (+1364090us) ts_tablet_manager.cc:1145] Deleting tablet cf8bb1c44a7b4d17bddeff8b2fe81b35
0810 08:00:35.929962 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:36.045471 (+115509us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:36.094088 (+ 48617us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:36.263518 (+169430us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1363538}
I20260810 08:00:36.265004 15478 tablet_replica.cc:330] T 924c0f73638f48a88c08d45d459132e1 P 3fc5e568f3724f158b616c303efb9eae: stopping tablet replica
I20260810 08:00:36.266793 15478 raft_consensus.cc:2229] T 924c0f73638f48a88c08d45d459132e1 P 3fc5e568f3724f158b616c303efb9eae [term 1 LEADER]: Raft consensus shutting down.
I20260810 08:00:36.267477 15478 pending_rounds.cc:62] T 924c0f73638f48a88c08d45d459132e1 P 3fc5e568f3724f158b616c303efb9eae: Trying to abort 1 pending ops.
I20260810 08:00:36.267680 15478 pending_rounds.cc:68] T 924c0f73638f48a88c08d45d459132e1 P 3fc5e568f3724f158b616c303efb9eae: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316884831329832960 op_type: NO_OP noop_request { }
I20260810 08:00:36.269105 15119 tablet_service.cc:1467] Processing CreateTablet for tablet fc0169c909454088b90ae8a5e01c1ec6 (DEFAULT_TABLE table=default.Table_93c38e9ecbdd4eeeb498ad5ebb5eef2b [id=28c79671b5a143c88c144dd6ea30bf88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:36.273082 15941 leader_election.cc:288] T a244ad3bda5d45f9b89f68b734338d21 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), a91b27762182454baada71e91ceab224 (127.14.57.193:43601)
I20260810 08:00:36.274451 15951 consensus_queue.cc:1035] T b5a0e0af742340b2945be2102768b307 P a91b27762182454baada71e91ceab224 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 08:00:36.274731 14989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a244ad3bda5d45f9b89f68b734338d21" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:36.276497 15951 raft_consensus.cc:482] T cb37a3a87f8b48a68c19510f306073c2 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:36.278833 15111 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ce493c78a30404ca15e5b2568446bdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:35 UTC) from {username='slave'} at 127.0.0.1:43002
W20260810 08:00:36.282081 15079 leader_election.cc:341] T a244ad3bda5d45f9b89f68b734338d21 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:36.284162 14965 tablet_service.cc:1467] Processing CreateTablet for tablet 0493eb96a2a042718527ddcd9524d799 (DEFAULT_TABLE table=default.Table_9a8eb842ebf244d5abbfb0f43373ca08 [id=23114ccea26d48079f910570c9580a71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:36.284533 15890 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34630 (request call id 145) took 2435 ms. Trace:
I20260810 08:00:36.285594 14965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0493eb96a2a042718527ddcd9524d799. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:36.286144 15118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f221dc79f1104fd0ab225c09da0df457. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:36.287458 15119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc0169c909454088b90ae8a5e01c1ec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:36.285063 15890 rpcz_store.cc:276] 0810 08:00:33.848999 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:33.849161 (+ 162us) service_pool.cc:227] Handling call
0810 08:00:35.679206 (+1830045us) ts_tablet_manager.cc:1145] Deleting tablet d0761b090d8147b8a5537b0ed51108a9
0810 08:00:35.679346 (+ 140us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:35.755084 (+ 75738us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:36.131195 (+376111us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:36.284404 (+153209us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1829286}
I20260810 08:00:36.294391 15890 tablet_replica.cc:330] T cb37a3a87f8b48a68c19510f306073c2 P a91b27762182454baada71e91ceab224: stopping tablet replica
I20260810 08:00:36.268023 15478 raft_consensus.cc:2866] T 924c0f73638f48a88c08d45d459132e1 P 3fc5e568f3724f158b616c303efb9eae [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260810 08:00:36.298836 15478 raft_consensus.cc:2258] T 924c0f73638f48a88c08d45d459132e1 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:36.303284 14689 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_a6db5abca8ce42899acc1f940ec59b5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:36.305585 15278 tablet_service.cc:1467] Processing CreateTablet for tablet fc0169c909454088b90ae8a5e01c1ec6 (DEFAULT_TABLE table=default.Table_93c38e9ecbdd4eeeb498ad5ebb5eef2b [id=28c79671b5a143c88c144dd6ea30bf88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:36.306694 15278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc0169c909454088b90ae8a5e01c1ec6. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 08:00:36.307439 14689 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a6db5abca8ce42899acc1f940ec59b5e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:36.313187 15267 tablet_service.cc:1514] Processing DeleteTablet for tablet ed8f3e52959b4a0c8ed2a053e05e5bd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:35 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:36.315809 15270 tablet_service.cc:1467] Processing CreateTablet for tablet 0493eb96a2a042718527ddcd9524d799 (DEFAULT_TABLE table=default.Table_9a8eb842ebf244d5abbfb0f43373ca08 [id=23114ccea26d48079f910570c9580a71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:36.317386 15270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0493eb96a2a042718527ddcd9524d799. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:36.317257 14695 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_a5c1e1f8311b4dc19e3d89b7baecd114" } modify_external_catalogs: true
I20260810 08:00:36.317871 14695 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_a5c1e1f8311b4dc19e3d89b7baecd114" } modify_external_catalogs: true
I20260810 08:00:36.330183 15275 tablet_service.cc:1467] Processing CreateTablet for tablet f221dc79f1104fd0ab225c09da0df457 (DEFAULT_TABLE table=default.Table_4ff94124f5cb4710a0ad3ff1781efcd3 [id=0b1956596de94b569ebd9f1816e4b699]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:36.303474 15302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68048e2768cf441d906e675254175c9c" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:36.348984 15275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f221dc79f1104fd0ab225c09da0df457. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:36.348335 15267 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ce493c78a30404ca15e5b2568446bdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:35 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:36.376947 14678 catalog_manager.cc:4841] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): tablet cf8bb1c44a7b4d17bddeff8b2fe81b35 (table default.Table_a6ce85a21a354e7c92f158c1ea3212b0 [id=8329ae05a9b0470897f5830376c1f87a]) successfully deleted
I20260810 08:00:36.382455 15267 tablet_service.cc:1467] Processing CreateTablet for tablet 5db943b869184e01994f0f84bb89a10a (DEFAULT_TABLE table=default.Table_e3bf1bdea6324b319c1a04b7231674f5 [id=20c0d637b1c34aff9e999f35d840f169]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:36.383383 15439 tablet_bootstrap.cc:492] T 561cbc6a4eb14d3584f88ea26a0fea7a P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:36.383800 15439 ts_tablet_manager.cc:1397] T 561cbc6a4eb14d3584f88ea26a0fea7a P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.450s user 0.004s sys 0.011s
I20260810 08:00:36.384999 15267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5db943b869184e01994f0f84bb89a10a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:36.385070 15478 ts_tablet_manager.cc:1905] T 924c0f73638f48a88c08d45d459132e1 P 3fc5e568f3724f158b616c303efb9eae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:36.400020 14945 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:34626 (request call id 39) took 1297 ms. Trace:
I20260810 08:00:36.400569 14945 rpcz_store.cc:276] 0810 08:00:34.985135 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:34.985344 (+ 209us) service_pool.cc:227] Handling call
0810 08:00:35.129526 (+144182us) spinlock_profiling.cc:243] Waited 2.58 ms on lock 0x7b5400120128. stack: 00007feaedce8033 00007feaedce7f30 00007feaed869534 00000000004db2b4 00000000004db249 00007feaf5004361 00007feaf50901f1 00007feaf5c556a0 00007feaf44fde04 00007feaf44fdd92 00007feaf44fdcc1 00007feaf44fdc3c 00007feaf44fcef2 00007feaf2a1e94c 00007feaf2a1dd86 00007feaf2a1d72f
0810 08:00:35.553956 (+424430us) spinlock_profiling.cc:243] Waited 420 ms on lock 0x7b54001202d4. stack: 00007feaedce8033 00007feaedce7f30 00007feaed869534 00000000004db2b4 00000000004db249 00007feaf4bf4471 00007feaf4c69268 00007feaf5c556cd 00007feaf44fde04 00007feaf44fdd92 00007feaf44fdcc1 00007feaf44fdc3c 00007feaf44fcef2 00007feaf2a1e94c 00007feaf2a1dd86 00007feaf2a1d72f
0810 08:00:35.685869 (+131913us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b54000e0954. stack: 00007feaedce8033 00007feaedce7f30 00007feaed869534 00000000004db2b4 00000000004db249 00007feaf4bf4471 00007feaf4c69268 00007feaf5c556cd 00007feaf44fde04 00007feaf44fdd92 00007feaf44fdcc1 00007feaf44fdc3c 00007feaf44fcef2 00007feaf2a1e94c 00007feaf2a1dd86 00007feaf2a1d72f
0810 08:00:35.852643 (+166774us) spinlock_profiling.cc:243] Waited 163 ms on lock 0x7b54000b27d4. stack: 00007feaedce8033 00007feaedce7f30 00007feaed869534 00000000004db2b4 00000000004db249 00007feaf4bf4471 00007feaf4c69268 00007feaf5c556cd 00007feaf44fde04 00007feaf44fdd92 00007feaf44fdcc1 00007feaf44fdc3c 00007feaf44fcef2 00007feaf2a1e94c 00007feaf2a1dd86 00007feaf2a1d72f
0810 08:00:36.282670 (+430027us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1634432512}
I20260810 08:00:36.409170 15302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a244ad3bda5d45f9b89f68b734338d21" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:36.415364 15437 tablet_bootstrap.cc:492] T a244ad3bda5d45f9b89f68b734338d21 P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:36.415807 14959 tablet_service.cc:1514] Processing DeleteTablet for tablet ed8f3e52959b4a0c8ed2a053e05e5bd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:35 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:36.415913 15437 ts_tablet_manager.cc:1397] T a244ad3bda5d45f9b89f68b734338d21 P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.590s user 0.021s sys 0.072s
I20260810 08:00:36.306432 15111 tablet_service.cc:1467] Processing CreateTablet for tablet 5db943b869184e01994f0f84bb89a10a (DEFAULT_TABLE table=default.Table_e3bf1bdea6324b319c1a04b7231674f5 [id=20c0d637b1c34aff9e999f35d840f169]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:36.340781 15479 ts_tablet_manager.cc:1918] T 8a0e1ad5653f46b4861b3e39629fbd2c P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:36.438046 15479 log.cc:1196] T 8a0e1ad5653f46b4861b3e39629fbd2c P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/8a0e1ad5653f46b4861b3e39629fbd2c
I20260810 08:00:36.277196 15951 raft_consensus.cc:504] T cb37a3a87f8b48a68c19510f306073c2 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:36.438048 14985 raft_consensus.cc:1261] T 1f427f4093e5417a9ca79b1805b0675e P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Refusing update from remote peer 3fc5e568f3724f158b616c303efb9eae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 08:00:36.438905 15479 ts_tablet_manager.cc:1939] T 8a0e1ad5653f46b4861b3e39629fbd2c P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
W20260810 08:00:36.373440 15076 leader_election.cc:341] T 68048e2768cf441d906e675254175c9c P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:36.439566 14956 tablet_service.cc:1467] Processing CreateTablet for tablet fc0169c909454088b90ae8a5e01c1ec6 (DEFAULT_TABLE table=default.Table_93c38e9ecbdd4eeeb498ad5ebb5eef2b [id=28c79671b5a143c88c144dd6ea30bf88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:36.440059 15076 leader_election.cc:302] T 68048e2768cf441d906e675254175c9c P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:36.441592 15941 raft_consensus.cc:2726] T 68048e2768cf441d906e675254175c9c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:36.442404 14956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc0169c909454088b90ae8a5e01c1ec6. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 08:00:36.442915 15076 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.195:37729, user_credentials={real_user=slave}} blocked reactor thread for 69539.9us
W20260810 08:00:36.448282 15076 leader_election.cc:341] T a244ad3bda5d45f9b89f68b734338d21 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:36.450783 15354 heartbeater.cc:503] Master 127.14.57.252:33819 was elected leader, sending a full tablet report...
I20260810 08:00:36.383396 15147 raft_consensus.cc:1261] T 1f427f4093e5417a9ca79b1805b0675e P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Refusing update from remote peer 3fc5e568f3724f158b616c303efb9eae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 08:00:36.454416 15945 consensus_queue.cc:1035] T 1f427f4093e5417a9ca79b1805b0675e P 3fc5e568f3724f158b616c303efb9eae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 08:00:36.453097 15076 leader_election.cc:302] T a244ad3bda5d45f9b89f68b734338d21 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:36.468773 15945 consensus_queue.cc:1035] T 1f427f4093e5417a9ca79b1805b0675e P 3fc5e568f3724f158b616c303efb9eae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 08:00:36.469671 15980 raft_consensus.cc:2726] T a244ad3bda5d45f9b89f68b734338d21 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 08:00:36.473060 14687 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ae51ddeb22fe4cf7a8aa2d8b2b70a535 consensus state has no leader
I20260810 08:00:36.478237 14946 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:34626 (request call id 38) took 1563 ms. Trace:
I20260810 08:00:36.478751 14946 rpcz_store.cc:276] 0810 08:00:34.841558 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:34.841767 (+ 209us) service_pool.cc:227] Handling call
0810 08:00:35.562271 (+720504us) spinlock_profiling.cc:243] Waited 483 ms on lock 0x7b54001202d4. stack: 00007feaedce8033 00007feaedce7f30 00007feaed869534 00000000004db2b4 00000000004db249 00007feaf4bf4471 00007feaf4c69268 00007feaf5c556cd 00007feaf44fde04 00007feaf44fdd92 00007feaf44fdcc1 00007feaf44fdc3c 00007feaf44fcef2 00007feaf2a1e94c 00007feaf2a1dd86 00007feaf2a1d72f
0810 08:00:35.595467 (+ 33196us) spinlock_profiling.cc:243] Waited 28.6 ms on lock 0x7b54000e0954. stack: 00007feaedce8033 00007feaedce7f30 00007feaed869534 00000000004db2b4 00000000004db249 00007feaf4bf4471 00007feaf4c69268 00007feaf5c556cd 00007feaf44fde04 00007feaf44fdd92 00007feaf44fdcc1 00007feaf44fdc3c 00007feaf44fcef2 00007feaf2a1e94c 00007feaf2a1dd86 00007feaf2a1d72f
0810 08:00:36.405137 (+809670us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1176789248}
I20260810 08:00:36.474037 15478 ts_tablet_manager.cc:1918] T 924c0f73638f48a88c08d45d459132e1 P 3fc5e568f3724f158b616c303efb9eae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 08:00:36.487504 15478 log.cc:1196] T 924c0f73638f48a88c08d45d459132e1 P 3fc5e568f3724f158b616c303efb9eae: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal/wals/924c0f73638f48a88c08d45d459132e1
I20260810 08:00:36.488265 15478 ts_tablet_manager.cc:1939] T 924c0f73638f48a88c08d45d459132e1 P 3fc5e568f3724f158b616c303efb9eae: Deleting consensus metadata
I20260810 08:00:36.489406 14681 catalog_manager.cc:4841] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): tablet d0761b090d8147b8a5537b0ed51108a9 (table default.Table_f387e847280a4f048905e15c5003f58c [id=fc7978f30c9a49c8aef649a6d10d87ad]) successfully deleted
I20260810 08:00:36.490247 14959 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ce493c78a30404ca15e5b2568446bdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:35 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:36.490664 14949 tablet_service.cc:1467] Processing CreateTablet for tablet 5db943b869184e01994f0f84bb89a10a (DEFAULT_TABLE table=default.Table_e3bf1bdea6324b319c1a04b7231674f5 [id=20c0d637b1c34aff9e999f35d840f169]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:36.498971 15111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5db943b869184e01994f0f84bb89a10a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:36.498685 14960 tablet_service.cc:1467] Processing CreateTablet for tablet f221dc79f1104fd0ab225c09da0df457 (DEFAULT_TABLE table=default.Table_4ff94124f5cb4710a0ad3ff1781efcd3 [id=0b1956596de94b569ebd9f1816e4b699]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:36.500062 15479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43002 (request call id 151) took 1453 ms. Trace:
I20260810 08:00:36.459913 15890 raft_consensus.cc:2229] T cb37a3a87f8b48a68c19510f306073c2 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:36.500442 15479 rpcz_store.cc:276] 0810 08:00:35.046504 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:35.046699 (+ 195us) service_pool.cc:227] Handling call
0810 08:00:36.144196 (+1097497us) ts_tablet_manager.cc:1145] Deleting tablet 8a0e1ad5653f46b4861b3e39629fbd2c
0810 08:00:36.144232 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:36.251980 (+107748us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:36.340561 (+ 88581us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:36.499999 (+159438us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31822,"tablet-delete.queue_time_us":1044492}
I20260810 08:00:36.501747 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet 8a0e1ad5653f46b4861b3e39629fbd2c (table default.Table_3d528c5b4aa54991bf128ada67924546 [id=8c493c9dfd3f4f358bc02f33869f3a16]) successfully deleted
I20260810 08:00:36.502205 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 158) took 1708 ms. Trace:
I20260810 08:00:36.502610 15478 rpcz_store.cc:276] 0810 08:00:34.794000 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:34.794212 (+ 212us) service_pool.cc:227] Handling call
0810 08:00:36.264859 (+1470647us) ts_tablet_manager.cc:1145] Deleting tablet 924c0f73638f48a88c08d45d459132e1
0810 08:00:36.264901 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:36.395965 (+131064us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:36.473814 (+ 77849us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:36.502075 (+ 28261us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":78961,"tablet-delete.queue_time_us":1469861}
W20260810 08:00:36.504007 14681 catalog_manager.cc:4821] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): delete failed for tablet abc607ab01f74b2c9c693d6dea73090c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abc607ab01f74b2c9c693d6dea73090c already in progress: creating tablet
I20260810 08:00:36.504148 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 159) took 1708 ms. Trace:
I20260810 08:00:36.504434 15478 rpcz_store.cc:276] 0810 08:00:34.795455 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:34.795604 (+ 149us) service_pool.cc:227] Handling call
0810 08:00:36.503797 (+1708193us) ts_tablet_manager.cc:1145] Deleting tablet 0a60001d76604d5085c843961d52213f
0810 08:00:36.503855 (+ 58us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:36.504094 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1707425}
W20260810 08:00:36.505761 14681 catalog_manager.cc:4808] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): delete failed for tablet fc0169c909454088b90ae8a5e01c1ec6 because the tablet was not found. No further retry: Not found: Tablet not found: fc0169c909454088b90ae8a5e01c1ec6
I20260810 08:00:36.505915 14678 catalog_manager.cc:4841] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): tablet 924c0f73638f48a88c08d45d459132e1 (table default.Table_409addd050ee45b981bddfacfab24c03 [id=76166bd7cb584067b7d2e305603a840b]) successfully deleted
I20260810 08:00:36.472352 15439 raft_consensus.cc:348] T 561cbc6a4eb14d3584f88ea26a0fea7a P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:36.506271 15439 raft_consensus.cc:374] T 561cbc6a4eb14d3584f88ea26a0fea7a P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260810 08:00:36.506536 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet 0a60001d76604d5085c843961d52213f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a60001d76604d5085c843961d52213f already in progress: creating tablet
I20260810 08:00:36.506765 15479 tablet_replica.cc:330] T 68048e2768cf441d906e675254175c9c P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
I20260810 08:00:36.507354 15280 tablet_service.cc:1467] Processing CreateTablet for tablet ed8f3e52959b4a0c8ed2a053e05e5bd6 (DEFAULT_TABLE table=default.Table_80fd70a445294836a1e5485e58cecde8 [id=fdd4857f284a46c2a1c29920fbfbf5db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:36.507560 15479 raft_consensus.cc:2229] T 68048e2768cf441d906e675254175c9c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:36.507972 15479 raft_consensus.cc:2258] T 68048e2768cf441d906e675254175c9c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:36.508315 15280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed8f3e52959b4a0c8ed2a053e05e5bd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:36.511435 15945 raft_consensus.cc:482] T cb37a3a87f8b48a68c19510f306073c2 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:36.511966 15945 raft_consensus.cc:504] T cb37a3a87f8b48a68c19510f306073c2 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:36.505468 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 160) took 1693 ms. Trace:
I20260810 08:00:36.514680 15110 tablet_service.cc:1467] Processing CreateTablet for tablet ed8f3e52959b4a0c8ed2a053e05e5bd6 (DEFAULT_TABLE table=default.Table_80fd70a445294836a1e5485e58cecde8 [id=fdd4857f284a46c2a1c29920fbfbf5db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:36.515552 15945 leader_election.cc:288] T cb37a3a87f8b48a68c19510f306073c2 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:36.518826 14687 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_814fa7cd026940c385ff0c1e0ec55a01" } modify_external_catalogs: true
I20260810 08:00:36.515726 15478 rpcz_store.cc:276] 0810 08:00:34.812319 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:34.812452 (+ 133us) service_pool.cc:227] Handling call
0810 08:00:36.505185 (+1692733us) ts_tablet_manager.cc:1145] Deleting tablet 5f2e51d10e7142938e03b2ef687b0edd
0810 08:00:36.505220 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:36.505416 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1691705}
I20260810 08:00:36.519541 14687 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_814fa7cd026940c385ff0c1e0ec55a01" } modify_external_catalogs: true
I20260810 08:00:36.520478 15478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 162) took 1612 ms. Trace:
I20260810 08:00:36.520773 15478 rpcz_store.cc:276] 0810 08:00:34.907829 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:34.908100 (+ 271us) service_pool.cc:227] Handling call
0810 08:00:36.520090 (+1611990us) ts_tablet_manager.cc:1145] Deleting tablet 521d16ada2424d3aa41f838a511131fb
0810 08:00:36.520132 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:36.520423 (+ 291us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1610900}
I20260810 08:00:36.522927 15960 tablet.cc:1722] T b5a0e0af742340b2945be2102768b307 P 9d7219686f304f62a54acc7497cbbd07: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260810 08:00:36.526573 15479 ts_tablet_manager.cc:1905] T 68048e2768cf441d906e675254175c9c P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:36.530570 15110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed8f3e52959b4a0c8ed2a053e05e5bd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:36.530555 15951 leader_election.cc:288] T cb37a3a87f8b48a68c19510f306073c2 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:36.544130 14688 master_service.cc:946] ReplaceTablet: received request to replace tablet 33cacf1276d44b24abe6aff12ea8ea51 from {username='slave'} at 127.0.0.1:42300
W20260810 08:00:36.553658 14918 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.57.195:37729, user_credentials={real_user=slave}} blocked reactor thread for 52687us
I20260810 08:00:36.566708 15890 raft_consensus.cc:2258] T cb37a3a87f8b48a68c19510f306073c2 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:36.567539 15965 tablet.cc:1722] T b5a0e0af742340b2945be2102768b307 P a91b27762182454baada71e91ceab224: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260810 08:00:36.575546 15437 raft_consensus.cc:348] T a244ad3bda5d45f9b89f68b734338d21 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:36.576285 15437 raft_consensus.cc:374] T a244ad3bda5d45f9b89f68b734338d21 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:36.576560 15437 raft_consensus.cc:729] T a244ad3bda5d45f9b89f68b734338d21 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:36.581399 15437 consensus_queue.cc:260] T a244ad3bda5d45f9b89f68b734338d21 P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:36.614396 15119 tablet_service.cc:1467] Processing CreateTablet for tablet 75681d70a3e742768df36a1eaacc3658 (DEFAULT_TABLE table=default.Table_97e91b68f06d4cca9ba065f2e6a07b0f [id=ea9c7a274180401c97a27049b434e7ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:36.619513 15437 ts_tablet_manager.cc:1428] T a244ad3bda5d45f9b89f68b734338d21 P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.203s user 0.006s sys 0.001s
I20260810 08:00:36.620978 15119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75681d70a3e742768df36a1eaacc3658. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:36.632825 15890 ts_tablet_manager.cc:1905] T cb37a3a87f8b48a68c19510f306073c2 P a91b27762182454baada71e91ceab224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:36.690670 14692 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_4361cdd1d34f4d8fa5a656a1b06218b0" } new_table_name: "default.Table_19d382281f324b2e82afd3247cebd644" modify_external_catalogs: true
I20260810 08:00:36.694460 14960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f221dc79f1104fd0ab225c09da0df457. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:36.700769 14949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5db943b869184e01994f0f84bb89a10a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:36.725648 15437 tablet_bootstrap.cc:492] T 5990e7ea243843bb869de5dde3e6cd23 P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:36.752046 15479 ts_tablet_manager.cc:1918] T 68048e2768cf441d906e675254175c9c P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:36.752485 15479 log.cc:1196] T 68048e2768cf441d906e675254175c9c P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/68048e2768cf441d906e675254175c9c
I20260810 08:00:36.753207 15479 ts_tablet_manager.cc:1939] T 68048e2768cf441d906e675254175c9c P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:36.759305 15490 tablet_bootstrap.cc:492] T 2fbda1b3b5b4482ba6f03f3e275d94e7 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:36.759770 15490 ts_tablet_manager.cc:1397] T 2fbda1b3b5b4482ba6f03f3e275d94e7 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.729s user 0.007s sys 0.019s
I20260810 08:00:36.776960 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet 68048e2768cf441d906e675254175c9c (table default.Table_4ff94124f5cb4710a0ad3ff1781efcd3 [id=0b1956596de94b569ebd9f1816e4b699]) successfully deleted
W20260810 08:00:36.778908 14681 catalog_manager.cc:4821] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): delete failed for tablet ed8f3e52959b4a0c8ed2a053e05e5bd6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed8f3e52959b4a0c8ed2a053e05e5bd6 already in progress: creating tablet
W20260810 08:00:36.780673 14681 catalog_manager.cc:4821] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): delete failed for tablet 1ce493c78a30404ca15e5b2568446bdf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ce493c78a30404ca15e5b2568446bdf already in progress: creating tablet
I20260810 08:00:36.798030 14953 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34630 (request call id 162) took 1032 ms. Trace:
I20260810 08:00:36.798720 14953 rpcz_store.cc:276] 0810 08:00:35.765435 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:35.766049 (+ 614us) service_pool.cc:227] Handling call
0810 08:00:35.767688 (+ 1639us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 08:00:35.767800 (+ 112us) ts_tablet_manager.cc:676] Creating new metadata...
0810 08:00:36.113979 (+346179us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:36.552736 (+438757us) tablet_replica.cc:205] Creating consensus instance
0810 08:00:36.797436 (+244700us) spinlock_profiling.cc:243] Waited 48.6 ms on lock 0x7b5800001898. stack: 00007feaedce8033 00007feaedce7f30 00007feaed869534 00000000004db2b4 00000000004db249 00007feaedb5ef11 00007feaedce2851 00007feaedce2e82 00007feaf6ba28b9 00007feaf5bd65d8 00007feaf5c9a063 00007feaf5c95e2f 00007feaf5c965ed 00007feaf5c47869 00007feaf4339c44 00007feaf4339bd2
0810 08:00:36.797938 (+ 502us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":111687424}
I20260810 08:00:36.807073 15437 tablet_bootstrap.cc:654] T 5990e7ea243843bb869de5dde3e6cd23 P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:36.812053 14694 master_service.cc:946] ReplaceTablet: received request to replace tablet 1f427f4093e5417a9ca79b1805b0675e from {username='slave'} at 127.0.0.1:42300
I20260810 08:00:36.818053 14690 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_d1f5fc54bbd148e28831f5512353d5f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:36.824445 14690 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1f5fc54bbd148e28831f5512353d5f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:36.831478 15490 raft_consensus.cc:348] T 2fbda1b3b5b4482ba6f03f3e275d94e7 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:36.832149 15490 raft_consensus.cc:374] T 2fbda1b3b5b4482ba6f03f3e275d94e7 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:36.832403 15490 raft_consensus.cc:729] T 2fbda1b3b5b4482ba6f03f3e275d94e7 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:36.833138 15490 consensus_queue.cc:260] T 2fbda1b3b5b4482ba6f03f3e275d94e7 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:36.850786 14686 catalog_manager.cc:6040] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: default.Table_4f3c632faa1e4c2d86acf10a656ee7c7 [id=fbd03ee96b004797a503841f4cc7249a] alter complete (version 1)
I20260810 08:00:36.506544 15439 raft_consensus.cc:729] T 561cbc6a4eb14d3584f88ea26a0fea7a P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:36.874284 14985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb37a3a87f8b48a68c19510f306073c2" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
W20260810 08:00:36.875483 14693 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.14.57.193:35941 (request call id 46) took 912 ms (client timeout 1000 ms). Trace:
W20260810 08:00:36.875895 14693 rpcz_store.cc:269] 0810 08:00:35.963501 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:35.965708 (+ 2207us) service_pool.cc:227] Handling call
0810 08:00:36.875376 (+909668us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
W20260810 08:00:36.883433 14922 leader_election.cc:341] T cb37a3a87f8b48a68c19510f306073c2 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Not found: Tablet not found: cb37a3a87f8b48a68c19510f306073c2
I20260810 08:00:36.885527 15490 ts_tablet_manager.cc:1428] T 2fbda1b3b5b4482ba6f03f3e275d94e7 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.125s user 0.006s sys 0.000s
I20260810 08:00:36.886704 15960 tablet.cc:1722] T b5a0e0af742340b2945be2102768b307 P 9d7219686f304f62a54acc7497cbbd07: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260810 08:00:36.892087 14953 tablet_service.cc:1467] Processing CreateTablet for tablet 75681d70a3e742768df36a1eaacc3658 (DEFAULT_TABLE table=default.Table_97e91b68f06d4cca9ba065f2e6a07b0f [id=ea9c7a274180401c97a27049b434e7ee]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 08:00:36.894500 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet 5f2e51d10e7142938e03b2ef687b0edd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f2e51d10e7142938e03b2ef687b0edd already in progress: creating tablet
W20260810 08:00:36.895430 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet 521d16ada2424d3aa41f838a511131fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 521d16ada2424d3aa41f838a511131fb already in progress: creating tablet
W20260810 08:00:36.896145 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet abc607ab01f74b2c9c693d6dea73090c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abc607ab01f74b2c9c693d6dea73090c already in progress: creating tablet
W20260810 08:00:36.896960 14678 catalog_manager.cc:4808] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet fc0169c909454088b90ae8a5e01c1ec6 because the tablet was not found. No further retry: Not found: Tablet not found: fc0169c909454088b90ae8a5e01c1ec6
I20260810 08:00:36.881399 14691 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_93c814deac714a3288f3980340acbc65" } new_table_name: "default.Table_a89c79d2a99b415cbac968e3e8a9d53d" modify_external_catalogs: true
W20260810 08:00:36.922022 14686 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.14.57.195:51883 (request call id 51) took 1041 ms (client timeout 1000 ms). Trace:
I20260810 08:00:36.931548 15490 tablet_bootstrap.cc:492] T a4a78b2a960d4d78aa1f48a9c9edf3be P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:36.936067 14953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75681d70a3e742768df36a1eaacc3658. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 08:00:36.938861 15970 lock_manager.cc:586] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 10664520935089164223) tablet 00000000000000000000000000000000 cur holder 0x7b5800040200
I20260810 08:00:36.939466 14954 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34630 (request call id 170) took 1014 ms. Trace:
I20260810 08:00:36.940178 14954 rpcz_store.cc:276] 0810 08:00:35.925462 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:35.953765 (+ 28303us) service_pool.cc:227] Handling call
0810 08:00:35.999738 (+ 45973us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 08:00:35.999824 (+ 86us) ts_tablet_manager.cc:676] Creating new metadata...
0810 08:00:36.643087 (+643263us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:36.902262 (+259175us) tablet_replica.cc:205] Creating consensus instance
0810 08:00:36.939364 (+ 37102us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 08:00:36.942119 14967 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34630 (request call id 166) took 1018 ms. Trace:
I20260810 08:00:36.942446 14967 rpcz_store.cc:276] 0810 08:00:35.924083 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:35.924296 (+ 213us) service_pool.cc:227] Handling call
0810 08:00:35.925826 (+ 1530us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 08:00:35.925908 (+ 82us) ts_tablet_manager.cc:676] Creating new metadata...
0810 08:00:36.754840 (+828932us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:36.915666 (+160826us) tablet_replica.cc:205] Creating consensus instance
0810 08:00:36.942062 (+ 26396us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 08:00:36.943169 15490 tablet_bootstrap.cc:654] T a4a78b2a960d4d78aa1f48a9c9edf3be P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
W20260810 08:00:36.952318 14922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.57.194:36265, user_credentials={real_user=slave}} blocked reactor thread for 47013.7us
W20260810 08:00:36.922969 14678 catalog_manager.cc:4808] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet ed8f3e52959b4a0c8ed2a053e05e5bd6 because the tablet was not found. No further retry: Not found: Tablet not found: ed8f3e52959b4a0c8ed2a053e05e5bd6
W20260810 08:00:36.957782 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet 1ce493c78a30404ca15e5b2568446bdf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ce493c78a30404ca15e5b2568446bdf already in progress: creating tablet
I20260810 08:00:36.974390 15302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb37a3a87f8b48a68c19510f306073c2" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:36.975271 15302 raft_consensus.cc:2454] T cb37a3a87f8b48a68c19510f306073c2 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 0.
I20260810 08:00:36.976750 15199 heartbeater.cc:347] Connected to a master server at 127.14.57.252:33819
W20260810 08:00:36.977195 14678 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 8b5a629a7f8041179a006fa412fb6b48 (table default.Table_a5c1e1f8311b4dc19e3d89b7baecd114 [id=b333dde37a3e456c9f5b361e5125c8cc], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8b5a629a7f8041179a006fa412fb6b48 consensus state has no leader
I20260810 08:00:36.981160 14952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34630 (request call id 171) took 1016 ms. Trace:
I20260810 08:00:36.991035 15490 tablet_bootstrap.cc:492] T a4a78b2a960d4d78aa1f48a9c9edf3be P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:36.991504 15490 ts_tablet_manager.cc:1397] T a4a78b2a960d4d78aa1f48a9c9edf3be P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.060s user 0.008s sys 0.011s
I20260810 08:00:36.981926 14952 rpcz_store.cc:276] 0810 08:00:35.964646 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:35.986046 (+ 21400us) service_pool.cc:227] Handling call
0810 08:00:35.989063 (+ 3017us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 08:00:35.989149 (+ 86us) ts_tablet_manager.cc:676] Creating new metadata...
0810 08:00:36.742197 (+753048us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:36.970857 (+228660us) tablet_replica.cc:205] Creating consensus instance
0810 08:00:36.981044 (+ 10187us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 08:00:37.004312 15965 tablet.cc:1722] T 5a22a48e2ba843b3a32a8900ab937ea0 P a91b27762182454baada71e91ceab224: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260810 08:00:37.014374 15490 raft_consensus.cc:348] T a4a78b2a960d4d78aa1f48a9c9edf3be P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:37.015228 15490 raft_consensus.cc:374] T a4a78b2a960d4d78aa1f48a9c9edf3be P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:37.019526 15490 raft_consensus.cc:729] T a4a78b2a960d4d78aa1f48a9c9edf3be P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:37.020520 15490 consensus_queue.cc:260] T a4a78b2a960d4d78aa1f48a9c9edf3be P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
W20260810 08:00:36.954947 14686 rpcz_store.cc:269] 0810 08:00:35.882663 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:35.883520 (+ 857us) service_pool.cc:227] Handling call
0810 08:00:36.921936 (+1038416us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 08:00:36.138128 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 08:00:36.138521 (+ 393us) write_op.cc:432] Acquiring schema lock in shared mode
0810 08:00:36.138558 (+ 37us) write_op.cc:435] Acquired schema lock
0810 08:00:36.138571 (+ 13us) tablet.cc:660] Decoding operations
0810 08:00:36.139433 (+ 862us) write_op.cc:620] Acquiring the partition lock for write op
0810 08:00:36.139484 (+ 51us) write_op.cc:641] Partition lock acquired for write op
0810 08:00:36.139506 (+ 22us) tablet.cc:683] Acquiring locks for 1 operations
0810 08:00:36.139725 (+ 219us) tablet.cc:699] Row locks acquired
0810 08:00:36.139740 (+ 15us) write_op.cc:260] PREPARE: finished
0810 08:00:36.173756 (+ 34016us) write_op.cc:270] Start()
0810 08:00:36.173855 (+ 99us) write_op.cc:276] Timestamp: P: 1786348836173653 usec, L: 0
0810 08:00:36.173873 (+ 18us) op_driver.cc:331] REPLICATION: starting
0810 08:00:36.174335 (+ 462us) log.cc:842] Serialized 533 byte log entry
0810 08:00:36.192423 (+ 18088us) op_driver.cc:444] REPLICATION: finished
0810 08:00:36.192707 (+ 284us) write_op.cc:301] APPLY: starting
0810 08:00:36.192808 (+ 101us) tablet.cc:1365] starting BulkCheckPresence
0810 08:00:36.192907 (+ 99us) tablet.cc:1368] finished BulkCheckPresence
0810 08:00:36.193084 (+ 177us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 08:00:36.193313 (+ 229us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 08:00:36.193582 (+ 269us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 08:00:36.193675 (+ 93us) write_op.cc:312] APPLY: finished
0810 08:00:36.193967 (+ 292us) log.cc:842] Serialized 23 byte log entry
0810 08:00:36.194253 (+ 286us) write_op.cc:489] Releasing partition, row and schema locks
0810 08:00:36.194347 (+ 94us) write_op.cc:454] Released schema lock
0810 08:00:36.194405 (+ 58us) write_op.cc:341] FINISH: Updating metrics
0810 08:00:36.194723 (+ 318us) write_op.cc:489] Releasing partition, row and schema locks
0810 08:00:36.194774 (+ 51us) write_op.cc:454] Released schema lock
Related trace 'op':
0810 08:00:36.781675 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 08:00:36.782370 (+ 695us) write_op.cc:432] Acquiring schema lock in shared mode
0810 08:00:36.782421 (+ 51us) write_op.cc:435] Acquired schema lock
0810 08:00:36.782436 (+ 15us) tablet.cc:660] Decoding operations
0810 08:00:36.834629 (+ 52193us) write_op.cc:620] Acquiring the partition lock for write op
0810 08:00:36.834671 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0810 08:00:36.834693 (+ 22us) tablet.cc:683] Acquiring locks for 1 operations
0810 08:00:36.834848 (+ 155us) tablet.cc:699] Row locks acquired
0810 08:00:36.834861 (+ 13us) write_op.cc:260] PREPARE: finished
0810 08:00:36.834975 (+ 114us) write_op.cc:270] Start()
0810 08:00:36.835056 (+ 81us) write_op.cc:276] Timestamp: P: 1786348836834933 usec, L: 0
0810 08:00:36.835072 (+ 16us) op_driver.cc:331] REPLICATION: starting
0810 08:00:36.835666 (+ 594us) log.cc:842] Serialized 356 byte log entry
0810 08:00:36.847230 (+ 11564us) op_driver.cc:444] REPLICATION: finished
0810 08:00:36.847718 (+ 488us) write_op.cc:301] APPLY: starting
0810 08:00:36.847819 (+ 101us) tablet.cc:1365] starting BulkCheckPresence
0810 08:00:36.848003 (+ 184us) tablet.cc:1368] finished BulkCheckPresence
0810 08:00:36.848406 (+ 403us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 08:00:36.848657 (+ 251us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 08:00:36.848933 (+ 276us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 08:00:36.849033 (+ 100us) write_op.cc:312] APPLY: finished
0810 08:00:36.849432 (+ 399us) log.cc:842] Serialized 23 byte log entry
0810 08:00:36.850287 (+ 855us) write_op.cc:489] Releasing partition, row and schema locks
0810 08:00:36.850407 (+ 120us) write_op.cc:454] Released schema lock
0810 08:00:36.850438 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0810 08:00:36.851672 (+ 1234us) write_op.cc:489] Releasing partition, row and schema locks
0810 08:00:36.851725 (+ 53us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":147,"apply.run_cpu_time_us":2310,"apply.run_wall_time_us":2316,"num_ops":1,"prepare.queue_time_us":243016,"prepare.run_cpu_time_us":2136,"prepare.run_wall_time_us":46730,"raft.queue_time_us":133,"raft.run_cpu_time_us":1139,"raft.run_wall_time_us":1938,"replication_time_us":18479}],["op",{"apply.queue_time_us":321,"apply.run_cpu_time_us":2613,"apply.run_wall_time_us":4286,"num_ops":1,"prepare.queue_time_us":582899,"prepare.run_cpu_time_us":2297,"prepare.run_wall_time_us":79538,"raft.queue_time_us":79,"raft.run_cpu_time_us":1187,"raft.run_wall_time_us":2114,"replication_time_us":12085}]]}
I20260810 08:00:37.030972 15490 ts_tablet_manager.cc:1428] T a4a78b2a960d4d78aa1f48a9c9edf3be P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.039s user 0.002s sys 0.011s
I20260810 08:00:37.032038 15490 tablet_bootstrap.cc:492] T 655fd33901974b9c85f52f92ff677247 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:37.065042 14918 leader_election.cc:302] T cb37a3a87f8b48a68c19510f306073c2 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224; no voters: 9d7219686f304f62a54acc7497cbbd07
W20260810 08:00:37.065841 14918 raft_consensus.cc:2670] T cb37a3a87f8b48a68c19510f306073c2 P a91b27762182454baada71e91ceab224: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260810 08:00:37.067440 14695 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet 8b5a629a7f8041179a006fa412fb6b48
I20260810 08:00:37.071321 15111 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b5a629a7f8041179a006fa412fb6b48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:36 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:37.071410 14956 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b5a629a7f8041179a006fa412fb6b48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:36 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:37.073845 16000 tablet.cc:1722] T b5a0e0af742340b2945be2102768b307 P 3fc5e568f3724f158b616c303efb9eae: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20260810 08:00:37.078532 14573 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.14.57.193:43601, user_credentials={real_user=slave}} blocked reactor thread for 484755us
W20260810 08:00:37.079330 14681 catalog_manager.cc:4821] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): delete failed for tablet 8b5a629a7f8041179a006fa412fb6b48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8b5a629a7f8041179a006fa412fb6b48 already in progress: creating tablet
W20260810 08:00:37.079511 15921 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.737s user 0.029s sys 0.066s
W20260810 08:00:37.079964 15921 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.738s user 0.029s sys 0.066s
I20260810 08:00:37.080708 15921 mvcc.cc:205] Tried to move back new op lower bound from 7316884833650339840 to 7316884830354616320. Current Snapshot: MvccSnapshot[applied={T|T < 7316884833650339840}]
I20260810 08:00:37.101348 15439 consensus_queue.cc:260] T 561cbc6a4eb14d3584f88ea26a0fea7a P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:37.103914 15439 ts_tablet_manager.cc:1428] T 561cbc6a4eb14d3584f88ea26a0fea7a P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.720s user 0.008s sys 0.000s
W20260810 08:00:37.104220 15439 ts_tablet_manager.cc:1460] T 561cbc6a4eb14d3584f88ea26a0fea7a P 3fc5e568f3724f158b616c303efb9eae: Tablet startup took 720ms
I20260810 08:00:37.112358 14695 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_60fc2701bb3a47e0bb362e24df4ff330"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:37.116686 14695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_60fc2701bb3a47e0bb362e24df4ff330 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:37.114424 14689 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_ac97594854254f2a9beb18b682277e3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:37.118036 14689 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac97594854254f2a9beb18b682277e3d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:37.111102 15951 raft_consensus.cc:482] T a244ad3bda5d45f9b89f68b734338d21 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:37.136248 15951 raft_consensus.cc:504] T a244ad3bda5d45f9b89f68b734338d21 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:37.145181 16001 tablet.cc:1717] T b5a0e0af742340b2945be2102768b307 P a91b27762182454baada71e91ceab224: Skipping requested alter to schema version 4, tablet already version 4
I20260810 08:00:37.146858 15300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a244ad3bda5d45f9b89f68b734338d21" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:37.147897 16001 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:34630 (request call id 173) took 1159 ms. Trace:
I20260810 08:00:37.149906 15146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a244ad3bda5d45f9b89f68b734338d21" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:37.151063 15146 raft_consensus.cc:2454] T a244ad3bda5d45f9b89f68b734338d21 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 0.
I20260810 08:00:37.132529 15995 tablet.cc:1722] T 1f427f4093e5417a9ca79b1805b0675e P 3fc5e568f3724f158b616c303efb9eae: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 08:00:37.158382 15490 tablet_bootstrap.cc:654] T 655fd33901974b9c85f52f92ff677247 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:37.157601 15992 tablet.cc:1722] T 5a22a48e2ba843b3a32a8900ab937ea0 P 9d7219686f304f62a54acc7497cbbd07: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20260810 08:00:37.163409 15234 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.57.194:36265, user_credentials={real_user=slave}} blocked reactor thread for 284554us
I20260810 08:00:37.164814 15278 tablet_service.cc:1514] Processing DeleteTablet for tablet 924c0f73638f48a88c08d45d459132e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 3281dd757ff44d9498ef70ad372eb150) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:37.168576 15275 tablet_service.cc:1514] Processing DeleteTablet for tablet cb37a3a87f8b48a68c19510f306073c2 with delete_type TABLET_DATA_DELETED (replaced by tablet b18980bffe6045b5b69bcecb47b4c711) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:37.170164 15275 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b5a629a7f8041179a006fa412fb6b48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:36 UTC) from {username='slave'} at 127.0.0.1:40692
W20260810 08:00:37.104490 15439 ts_tablet_manager.cc:1462] T 561cbc6a4eb14d3584f88ea26a0fea7a P 3fc5e568f3724f158b616c303efb9eae: Trace:
0810 08:00:27.702142 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:27.702252 (+ 110us) service_pool.cc:227] Handling call
0810 08:00:27.709758 (+ 7506us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 08:00:27.709891 (+ 133us) ts_tablet_manager.cc:676] Creating new metadata...
0810 08:00:28.053685 (+343794us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:28.138858 (+ 85173us) tablet_replica.cc:205] Creating consensus instance
0810 08:00:28.139301 (+ 443us) inbound_call.cc:173] Queueing success response
0810 08:00:35.933989 (+7794688us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0810 08:00:36.384041 (+450052us) ts_tablet_manager.cc:1429] Starting tablet replica
0810 08:00:36.448108 (+ 64067us) tablet_replica.cc:259] Starting instrumentation
0810 08:00:36.448729 (+ 621us) tablet_replica.cc:278] Starting consensus
Metrics: {"tablet-open.queue_time_us":7794768}
W20260810 08:00:37.176944 15234 leader_election.cc:341] T cb37a3a87f8b48a68c19510f306073c2 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: Tablet not RUNNING: STOPPED
W20260810 08:00:37.196553 15234 leader_election.cc:341] T cb37a3a87f8b48a68c19510f306073c2 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Not found: Tablet not found: cb37a3a87f8b48a68c19510f306073c2
I20260810 08:00:37.197911 15234 leader_election.cc:302] T cb37a3a87f8b48a68c19510f306073c2 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae; no voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224
I20260810 08:00:37.202980 15997 raft_consensus.cc:2726] T cb37a3a87f8b48a68c19510f306073c2 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:37.205974 14687 catalog_manager.cc:5777] Not sending DeleteTablet requests; no consensus state for tablet 175af9840d564708a5dbb5cffc94d3f7
I20260810 08:00:37.206569 14922 leader_election.cc:302] T a244ad3bda5d45f9b89f68b734338d21 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224; no voters:
I20260810 08:00:37.210368 15270 tablet_service.cc:1467] Processing CreateTablet for tablet 75681d70a3e742768df36a1eaacc3658 (DEFAULT_TABLE table=default.Table_97e91b68f06d4cca9ba065f2e6a07b0f [id=ea9c7a274180401c97a27049b434e7ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:37.211369 15980 raft_consensus.cc:482] T e0652a7684b04a89840049ce0d781199 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:37.212343 15270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75681d70a3e742768df36a1eaacc3658. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:37.212109 15980 raft_consensus.cc:504] T e0652a7684b04a89840049ce0d781199 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:37.200547 15439 tablet_bootstrap.cc:492] T a244ad3bda5d45f9b89f68b734338d21 P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:37.217980 15997 raft_consensus.cc:482] T 561cbc6a4eb14d3584f88ea26a0fea7a P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:37.218777 15997 raft_consensus.cc:504] T 561cbc6a4eb14d3584f88ea26a0fea7a P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:37.222350 15980 leader_election.cc:288] T e0652a7684b04a89840049ce0d781199 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:37.223701 15980 raft_consensus.cc:482] T a4a78b2a960d4d78aa1f48a9c9edf3be P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:37.225212 15980 raft_consensus.cc:504] T a4a78b2a960d4d78aa1f48a9c9edf3be P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:37.229214 15997 leader_election.cc:288] T 561cbc6a4eb14d3584f88ea26a0fea7a P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
W20260810 08:00:37.238674 14692 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8e6236e9f37146f6956e7a46eb3df3d5 consensus state has no leader
I20260810 08:00:37.241808 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0652a7684b04a89840049ce0d781199" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:37.243997 16000 tablet.cc:1722] T 5a22a48e2ba843b3a32a8900ab937ea0 P 3fc5e568f3724f158b616c303efb9eae: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260810 08:00:37.249269 15146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561cbc6a4eb14d3584f88ea26a0fea7a" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:37.249840 15980 leader_election.cc:288] T a4a78b2a960d4d78aa1f48a9c9edf3be P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:37.254319 14688 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet 33cacf1276d44b24abe6aff12ea8ea51
W20260810 08:00:37.254419 15076 leader_election.cc:341] T e0652a7684b04a89840049ce0d781199 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:37.255826 14688 catalog_manager.cc:6565] ReplaceTablet: tablet 33cacf1276d44b24abe6aff12ea8ea51 deleted and replaced by tablet 04e61f387c794c4491752022c5ca0c8a
I20260810 08:00:37.258803 15111 tablet_service.cc:1514] Processing DeleteTablet for tablet 33cacf1276d44b24abe6aff12ea8ea51 with delete_type TABLET_DATA_DELETED (replaced by tablet 04e61f387c794c4491752022c5ca0c8a) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:37.261044 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4a78b2a960d4d78aa1f48a9c9edf3be" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:37.262508 15267 tablet_service.cc:1514] Processing DeleteTablet for tablet 33cacf1276d44b24abe6aff12ea8ea51 with delete_type TABLET_DATA_DELETED (replaced by tablet 04e61f387c794c4491752022c5ca0c8a) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:37.264814 15439 tablet_bootstrap.cc:654] T a244ad3bda5d45f9b89f68b734338d21 P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
W20260810 08:00:37.270643 15076 leader_election.cc:341] T a4a78b2a960d4d78aa1f48a9c9edf3be P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:37.282053 15980 raft_consensus.cc:482] T e13b91d61063446f9d1454da4f8bb706 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:37.282621 15980 raft_consensus.cc:504] T e13b91d61063446f9d1454da4f8bb706 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:37.148030 15951 leader_election.cc:288] T a244ad3bda5d45f9b89f68b734338d21 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:37.285873 15980 leader_election.cc:288] T e13b91d61063446f9d1454da4f8bb706 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:37.287106 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13b91d61063446f9d1454da4f8bb706" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:37.288033 15490 tablet_bootstrap.cc:492] T 655fd33901974b9c85f52f92ff677247 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:37.288425 15490 ts_tablet_manager.cc:1397] T 655fd33901974b9c85f52f92ff677247 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.257s user 0.003s sys 0.015s
I20260810 08:00:37.292213 15490 raft_consensus.cc:348] T 655fd33901974b9c85f52f92ff677247 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:37.292922 15490 raft_consensus.cc:374] T 655fd33901974b9c85f52f92ff677247 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:37.293138 15490 raft_consensus.cc:729] T 655fd33901974b9c85f52f92ff677247 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:37.293767 15490 consensus_queue.cc:260] T 655fd33901974b9c85f52f92ff677247 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:37.295836 15490 ts_tablet_manager.cc:1428] T 655fd33901974b9c85f52f92ff677247 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260810 08:00:37.148627 16001 rpcz_store.cc:276] 0810 08:00:35.988189 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:35.990847 (+ 2658us) service_pool.cc:227] Handling call
0810 08:00:37.147750 (+1156903us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 08:00:36.279706 (+ 0us) alter_schema_op.cc:102] PREPARE ALTER-SCHEMA: Starting
0810 08:00:36.280309 (+ 603us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0810 08:00:36.763807 (+483498us) alter_schema_op.cc:65] Acquired schema lock
0810 08:00:36.763895 (+ 88us) alter_schema_op.cc:117] PREPARE ALTER-SCHEMA: finished
0810 08:00:36.764028 (+ 133us) alter_schema_op.cc:126] START. Timestamp: 1786348836763797
0810 08:00:36.764051 (+ 23us) op_driver.cc:331] REPLICATION: starting
0810 08:00:36.764582 (+ 531us) log.cc:842] Serialized 173 byte log entry
0810 08:00:37.102012 (+337430us) op_driver.cc:444] REPLICATION: finished
0810 08:00:37.144858 (+ 42846us) alter_schema_op.cc:131] APPLY ALTER-SCHEMA: Starting
0810 08:00:37.146486 (+ 1628us) log.cc:842] Serialized 94 byte log entry
0810 08:00:37.147553 (+ 1067us) alter_schema_op.cc:71] Released schema lock
0810 08:00:37.147583 (+ 30us) alter_schema_op.cc:184] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":42587,"prepare.queue_time_us":287218,"prepare.run_cpu_time_us":24077,"prepare.run_wall_time_us":485078,"replication_time_us":337799,"thread_start_us":33231,"threads_started":1}]]}
I20260810 08:00:37.137190 14955 tablet_service.cc:1467] Processing CreateTablet for tablet ed8f3e52959b4a0c8ed2a053e05e5bd6 (DEFAULT_TABLE table=default.Table_80fd70a445294836a1e5485e58cecde8 [id=fdd4857f284a46c2a1c29920fbfbf5db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:37.331791 14955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed8f3e52959b4a0c8ed2a053e05e5bd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:37.299206 15890 ts_tablet_manager.cc:1918] T cb37a3a87f8b48a68c19510f306073c2 P a91b27762182454baada71e91ceab224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:37.337257 15890 log.cc:1196] T cb37a3a87f8b48a68c19510f306073c2 P a91b27762182454baada71e91ceab224: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal/wals/cb37a3a87f8b48a68c19510f306073c2
I20260810 08:00:37.338055 15890 ts_tablet_manager.cc:1939] T cb37a3a87f8b48a68c19510f306073c2 P a91b27762182454baada71e91ceab224: Deleting consensus metadata
I20260810 08:00:37.340672 15890 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34630 (request call id 146) took 3490 ms. Trace:
I20260810 08:00:37.286022 15988 raft_consensus.cc:2781] T a244ad3bda5d45f9b89f68b734338d21 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:37.342000 15988 raft_consensus.cc:482] T a244ad3bda5d45f9b89f68b734338d21 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:37.342361 15988 raft_consensus.cc:3037] T a244ad3bda5d45f9b89f68b734338d21 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:37.345930 15490 tablet_bootstrap.cc:492] T 0687d323788642f798465ece973702bb P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:37.291980 14949 tablet_service.cc:1514] Processing DeleteTablet for tablet 33cacf1276d44b24abe6aff12ea8ea51 with delete_type TABLET_DATA_DELETED (replaced by tablet 04e61f387c794c4491752022c5ca0c8a) from {username='slave'} at 127.0.0.1:34630
W20260810 08:00:37.288857 14918 leader_election.cc:341] T a244ad3bda5d45f9b89f68b734338d21 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
W20260810 08:00:37.347208 14918 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.195:37729, user_credentials={real_user=slave}} blocked reactor thread for 61026.6us
I20260810 08:00:37.350262 14989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4a78b2a960d4d78aa1f48a9c9edf3be" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
W20260810 08:00:37.352591 14681 catalog_manager.cc:4821] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): delete failed for tablet 33cacf1276d44b24abe6aff12ea8ea51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33cacf1276d44b24abe6aff12ea8ea51 already in progress: creating tablet
I20260810 08:00:37.352954 14988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0652a7684b04a89840049ce0d781199" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:37.357366 15490 tablet_bootstrap.cc:654] T 0687d323788642f798465ece973702bb P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:37.359877 15301 raft_consensus.cc:3037] T 561cbc6a4eb14d3584f88ea26a0fea7a P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:37.362815 14989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13b91d61063446f9d1454da4f8bb706" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:37.365103 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561cbc6a4eb14d3584f88ea26a0fea7a" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:37.367403 16003 tablet.cc:1717] T b5a0e0af742340b2945be2102768b307 P 9d7219686f304f62a54acc7497cbbd07: Skipping requested alter to schema version 4, tablet already version 4
I20260810 08:00:37.374894 14694 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet 1f427f4093e5417a9ca79b1805b0675e
I20260810 08:00:37.376629 14694 catalog_manager.cc:6565] ReplaceTablet: tablet 1f427f4093e5417a9ca79b1805b0675e deleted and replaced by tablet a58b33d9f4584a5294ecfbb092a2a6a1
I20260810 08:00:37.378311 14949 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f427f4093e5417a9ca79b1805b0675e with delete_type TABLET_DATA_DELETED (replaced by tablet a58b33d9f4584a5294ecfbb092a2a6a1) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:37.380692 15111 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f427f4093e5417a9ca79b1805b0675e with delete_type TABLET_DATA_DELETED (replaced by tablet a58b33d9f4584a5294ecfbb092a2a6a1) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:37.381793 15479 tablet_replica.cc:330] T 1f427f4093e5417a9ca79b1805b0675e P 9d7219686f304f62a54acc7497cbbd07: stopping tablet replica
W20260810 08:00:37.388844 15079 leader_election.cc:341] T a4a78b2a960d4d78aa1f48a9c9edf3be P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:37.389274 15079 leader_election.cc:302] T a4a78b2a960d4d78aa1f48a9c9edf3be P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:37.390019 15980 raft_consensus.cc:2726] T a4a78b2a960d4d78aa1f48a9c9edf3be P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 08:00:37.391011 15079 leader_election.cc:341] T e0652a7684b04a89840049ce0d781199 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:37.391462 15079 leader_election.cc:302] T e0652a7684b04a89840049ce0d781199 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:37.392483 15980 raft_consensus.cc:2726] T e0652a7684b04a89840049ce0d781199 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 08:00:37.409262 15079 leader_election.cc:341] T e13b91d61063446f9d1454da4f8bb706 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
W20260810 08:00:37.414327 14691 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d25928e4c39d45e69c3efeb36f03aece consensus state has no leader
I20260810 08:00:37.340996 15890 rpcz_store.cc:276] 0810 08:00:33.850624 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:33.850758 (+ 134us) service_pool.cc:227] Handling call
0810 08:00:36.294064 (+2443306us) ts_tablet_manager.cc:1145] Deleting tablet cb37a3a87f8b48a68c19510f306073c2
0810 08:00:36.294216 (+ 152us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:36.538821 (+244605us) spinlock_profiling.cc:243] Waited 164 ms on lock 0x7b5400110654. stack: 00007feaedce8033 00007feaedce7f30 00007feaed869534 00000000004db2b4 00000000004db249 00007feaf4bf4471 00007feaf4c77352 00007feaf508d26c 00007feaf5c9acfa 00007feaf5ccae10 00007feaf5cb1e0d 00007feaf5cb1db9 00007feaf5cb1d49 00007feaf5cb1d11 00007feaf5cb0f3d 0000000000580784
0810 08:00:36.918671 (+379850us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:37.298807 (+380136us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:37.340604 (+ 41797us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18927,"spinlock_wait_cycles":376347520,"tablet-delete.queue_time_us":2442746}
I20260810 08:00:37.425719 14690 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_1939fe86b63e4117be863f56bfe7131b" } modify_external_catalogs: true
I20260810 08:00:37.425932 15890 tablet_replica.cc:330] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P a91b27762182454baada71e91ceab224: stopping tablet replica
I20260810 08:00:37.426352 14690 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_1939fe86b63e4117be863f56bfe7131b" } modify_external_catalogs: true
I20260810 08:00:37.427443 15890 raft_consensus.cc:2229] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:37.428177 14681 catalog_manager.cc:4841] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): tablet cb37a3a87f8b48a68c19510f306073c2 (table default.Table_64e1c230e9624fc9ba6c328346150ffe [id=46e6c2115d8c4b5e9907ed76b273638a]) successfully deleted
I20260810 08:00:37.432003 16002 tablet_replica.cc:330] T cb37a3a87f8b48a68c19510f306073c2 P 3fc5e568f3724f158b616c303efb9eae: stopping tablet replica
I20260810 08:00:37.432927 16002 raft_consensus.cc:2229] T cb37a3a87f8b48a68c19510f306073c2 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:37.437320 14688 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_ccb04058c52049ba833379f6cdbb9df2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:37.438762 15980 raft_consensus.cc:482] T 8e603d04f7454501b2edfb5a1f4ea7f0 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:37.439281 15980 raft_consensus.cc:504] T 8e603d04f7454501b2edfb5a1f4ea7f0 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:37.442041 14692 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_97e91b68f06d4cca9ba065f2e6a07b0f" } modify_external_catalogs: true
I20260810 08:00:37.442565 14692 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_97e91b68f06d4cca9ba065f2e6a07b0f" } modify_external_catalogs: true
I20260810 08:00:37.442757 15980 leader_election.cc:288] T 8e603d04f7454501b2edfb5a1f4ea7f0 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
W20260810 08:00:37.438985 14688 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ccb04058c52049ba833379f6cdbb9df2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:37.449076 15980 raft_consensus.cc:482] T 8a8454f693494ab3b82f5b85c2d62bb2 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:37.449487 15980 raft_consensus.cc:504] T 8a8454f693494ab3b82f5b85c2d62bb2 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:37.451802 15980 leader_election.cc:288] T 8a8454f693494ab3b82f5b85c2d62bb2 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:37.452798 14985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e603d04f7454501b2edfb5a1f4ea7f0" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
W20260810 08:00:37.453974 15079 leader_election.cc:341] T 8e603d04f7454501b2edfb5a1f4ea7f0 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:37.455364 14985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8454f693494ab3b82f5b85c2d62bb2" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
W20260810 08:00:37.469302 15079 leader_election.cc:341] T 8a8454f693494ab3b82f5b85c2d62bb2 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:37.471930 15479 raft_consensus.cc:2229] T 1f427f4093e5417a9ca79b1805b0675e P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:37.472939 15479 raft_consensus.cc:2258] T 1f427f4093e5417a9ca79b1805b0675e P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:37.475241 15960 tablet.cc:1722] T 1f427f4093e5417a9ca79b1805b0675e P 9d7219686f304f62a54acc7497cbbd07: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260810 08:00:37.485543 15076 leader_election.cc:341] T e13b91d61063446f9d1454da4f8bb706 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:37.486255 15076 leader_election.cc:302] T e13b91d61063446f9d1454da4f8bb706 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:37.487556 15980 raft_consensus.cc:2726] T e13b91d61063446f9d1454da4f8bb706 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:37.448647 16000 tablet.cc:1717] T b5a0e0af742340b2945be2102768b307 P 3fc5e568f3724f158b616c303efb9eae: Skipping requested alter to schema version 4, tablet already version 4
I20260810 08:00:37.474351 15439 tablet_bootstrap.cc:492] T a244ad3bda5d45f9b89f68b734338d21 P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:37.490543 15439 ts_tablet_manager.cc:1397] T a244ad3bda5d45f9b89f68b734338d21 P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.290s user 0.006s sys 0.011s
I20260810 08:00:37.493201 15890 pending_rounds.cc:62] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P a91b27762182454baada71e91ceab224: Trying to abort 1 pending ops.
I20260810 08:00:37.493753 15890 pending_rounds.cc:68] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P a91b27762182454baada71e91ceab224: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316884826890682368 op_type: NO_OP noop_request { }
I20260810 08:00:37.494576 15890 raft_consensus.cc:2866] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
W20260810 08:00:37.498224 14678 catalog_manager.cc:4808] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet 924c0f73638f48a88c08d45d459132e1 because the tablet was not found. No further retry: Not found: Tablet not found: 924c0f73638f48a88c08d45d459132e1
I20260810 08:00:37.502493 14695 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_e6b03850596042a6a30ee45ea0720b86" } new_table_name: "default.Table_4a73c802a93940dea0793cc5e71e1f6f" modify_external_catalogs: true
I20260810 08:00:37.523962 15490 tablet_bootstrap.cc:492] T 0687d323788642f798465ece973702bb P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:37.524438 15490 ts_tablet_manager.cc:1397] T 0687d323788642f798465ece973702bb P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.179s user 0.013s sys 0.002s
W20260810 08:00:37.529251 14695 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 389b30cba2b149e9ab0de8c191808db2 consensus state has no leader
I20260810 08:00:37.529364 15890 raft_consensus.cc:2258] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:37.533058 15890 ts_tablet_manager.cc:1905] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P a91b27762182454baada71e91ceab224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:37.544343 16002 raft_consensus.cc:2258] T cb37a3a87f8b48a68c19510f306073c2 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Raft consensus is shut down!
W20260810 08:00:37.545521 15234 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.57.194:36265, user_credentials={real_user=slave}} blocked reactor thread for 47395.7us
I20260810 08:00:37.546470 16002 ts_tablet_manager.cc:1905] T cb37a3a87f8b48a68c19510f306073c2 P 3fc5e568f3724f158b616c303efb9eae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:37.547108 15299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e603d04f7454501b2edfb5a1f4ea7f0" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:37.549901 15299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8454f693494ab3b82f5b85c2d62bb2" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:37.553736 15234 leader_election.cc:302] T 561cbc6a4eb14d3584f88ea26a0fea7a P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae; no voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224
I20260810 08:00:37.569708 15490 raft_consensus.cc:348] T 0687d323788642f798465ece973702bb P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:37.570416 15490 raft_consensus.cc:374] T 0687d323788642f798465ece973702bb P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:37.570662 15490 raft_consensus.cc:729] T 0687d323788642f798465ece973702bb P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:37.576473 15439 raft_consensus.cc:348] T a244ad3bda5d45f9b89f68b734338d21 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:37.577486 15439 raft_consensus.cc:374] T a244ad3bda5d45f9b89f68b734338d21 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:37.577857 15439 raft_consensus.cc:729] T a244ad3bda5d45f9b89f68b734338d21 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:37.578739 15439 consensus_queue.cc:260] T a244ad3bda5d45f9b89f68b734338d21 P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:37.582095 15439 ts_tablet_manager.cc:1428] T a244ad3bda5d45f9b89f68b734338d21 P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.091s user 0.007s sys 0.000s
I20260810 08:00:37.505462 15965 tablet.cc:1722] T 1f427f4093e5417a9ca79b1805b0675e P a91b27762182454baada71e91ceab224: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 08:00:37.524619 15437 tablet_bootstrap.cc:492] T 5990e7ea243843bb869de5dde3e6cd23 P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:37.599704 15437 ts_tablet_manager.cc:1397] T 5990e7ea243843bb869de5dde3e6cd23 P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.874s user 0.011s sys 0.015s
I20260810 08:00:37.605700 14695 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_cb30530bbc01408888f095d0271f60c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:37.607275 14695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb30530bbc01408888f095d0271f60c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:37.613257 15980 raft_consensus.cc:482] T 2fbda1b3b5b4482ba6f03f3e275d94e7 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:37.613999 15980 raft_consensus.cc:504] T 2fbda1b3b5b4482ba6f03f3e275d94e7 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
W20260810 08:00:37.625670 16015 log.cc:924] Time spent T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be: Append to log took a long time: real 0.061s user 0.016s sys 0.004s
I20260810 08:00:37.625761 14688 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_f8637789a86d4e0da9392352f395f0c3" } new_table_name: "default.Table_8dd0e3d12b2c461dbedf3a2ac454cf93" modify_external_catalogs: true
I20260810 08:00:37.635279 15992 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:43002 (request call id 175) took 1012 ms. Trace:
I20260810 08:00:37.637925 15490 consensus_queue.cc:260] T 0687d323788642f798465ece973702bb P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:37.652206 15980 leader_election.cc:288] T 2fbda1b3b5b4482ba6f03f3e275d94e7 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
W20260810 08:00:37.656904 14688 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4ca688b4031c41a9884829fbd0be1f33 consensus state has no leader
I20260810 08:00:37.666857 15437 raft_consensus.cc:348] T 5990e7ea243843bb869de5dde3e6cd23 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:37.667896 15490 ts_tablet_manager.cc:1428] T 0687d323788642f798465ece973702bb P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.143s user 0.008s sys 0.003s
I20260810 08:00:37.585911 15278 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f427f4093e5417a9ca79b1805b0675e with delete_type TABLET_DATA_DELETED (replaced by tablet a58b33d9f4584a5294ecfbb092a2a6a1) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:37.588440 15354 heartbeater.cc:503] Master 127.14.57.252:33819 was elected leader, sending a full tablet report...
I20260810 08:00:37.674633 15439 tablet_bootstrap.cc:492] T 8364045e3fa2441e8dc1a1d87f73361d P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:37.635732 15992 rpcz_store.cc:276] 0810 08:00:36.623222 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:36.625711 (+ 2489us) service_pool.cc:227] Handling call
0810 08:00:37.635197 (+1009486us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 08:00:36.631862 (+ 0us) alter_schema_op.cc:102] PREPARE ALTER-SCHEMA: Starting
0810 08:00:36.632455 (+ 593us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0810 08:00:36.632512 (+ 57us) alter_schema_op.cc:65] Acquired schema lock
0810 08:00:36.632581 (+ 69us) alter_schema_op.cc:117] PREPARE ALTER-SCHEMA: finished
0810 08:00:36.632793 (+ 212us) alter_schema_op.cc:126] START. Timestamp: 1786348836632627
0810 08:00:36.632844 (+ 51us) op_driver.cc:331] REPLICATION: starting
0810 08:00:36.633918 (+ 1074us) log.cc:842] Serialized 173 byte log entry
0810 08:00:36.892480 (+258562us) op_driver.cc:444] REPLICATION: finished
0810 08:00:37.157315 (+264835us) alter_schema_op.cc:131] APPLY ALTER-SCHEMA: Starting
0810 08:00:37.172446 (+ 15131us) log.cc:842] Serialized 5 byte log entry
0810 08:00:37.567318 (+394872us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:37.633887 (+ 66569us) log.cc:842] Serialized 14 byte log entry
0810 08:00:37.634978 (+ 1091us) alter_schema_op.cc:71] Released schema lock
0810 08:00:37.635034 (+ 56us) alter_schema_op.cc:184] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":264516,"prepare.queue_time_us":4567,"prepare.run_cpu_time_us":26743,"prepare.run_wall_time_us":132623,"raft.queue_time_us":102,"raft.run_cpu_time_us":850,"raft.run_wall_time_us":850,"replication_time_us":259508,"thread_start_us":366968,"threads_started":2,"wal-append.queue_time_us":121513}]]}
I20260810 08:00:37.504359 15988 raft_consensus.cc:504] T a244ad3bda5d45f9b89f68b734338d21 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:37.669219 15490 tablet_bootstrap.cc:492] T e1b42eac370145d4b684456639968485 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:37.694844 15479 ts_tablet_manager.cc:1905] T 1f427f4093e5417a9ca79b1805b0675e P 9d7219686f304f62a54acc7497cbbd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:37.701192 15299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a244ad3bda5d45f9b89f68b734338d21" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae"
I20260810 08:00:37.702185 15299 raft_consensus.cc:3037] T a244ad3bda5d45f9b89f68b734338d21 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:37.673195 14988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fbda1b3b5b4482ba6f03f3e275d94e7" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:37.709304 15299 raft_consensus.cc:2454] T a244ad3bda5d45f9b89f68b734338d21 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 1.
W20260810 08:00:37.709930 15234 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.14.57.252:33819, user_credentials={real_user=slave}} blocked reactor thread for 127060us
W20260810 08:00:37.723182 15076 leader_election.cc:341] T 8e603d04f7454501b2edfb5a1f4ea7f0 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:37.723621 15076 leader_election.cc:302] T 8e603d04f7454501b2edfb5a1f4ea7f0 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:37.725041 15997 raft_consensus.cc:2726] T 561cbc6a4eb14d3584f88ea26a0fea7a P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:37.725937 15980 raft_consensus.cc:2726] T 8e603d04f7454501b2edfb5a1f4ea7f0 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 08:00:37.727586 15076 leader_election.cc:341] T 8a8454f693494ab3b82f5b85c2d62bb2 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:37.727977 15076 leader_election.cc:302] T 8a8454f693494ab3b82f5b85c2d62bb2 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:37.729244 15980 raft_consensus.cc:2726] T 8a8454f693494ab3b82f5b85c2d62bb2 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:37.735937 14695 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_422ca616926440fcbb52753a5806c225"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:37.738965 14695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_422ca616926440fcbb52753a5806c225 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:37.740952 14688 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_697df159d8d14451b4f7beeb52f9e54a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:37.742218 14691 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_d81fe7cc77ad45d39c8b44e1299dbefc" } new_table_name: "default.Table_37cc86e3e13c4814818c22b78793d05c" modify_external_catalogs: true
I20260810 08:00:37.740845 14689 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_8caff4077c034aa69c250de0dd3f1a71"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 08:00:37.743561 15299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fbda1b3b5b4482ba6f03f3e275d94e7" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
W20260810 08:00:37.743958 14689 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8caff4077c034aa69c250de0dd3f1a71 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 08:00:37.745266 15076 leader_election.cc:341] T 2fbda1b3b5b4482ba6f03f3e275d94e7 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:37.752527 14693 catalog_manager.cc:5587] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265) does not have the latest schema for tablet b5a0e0af742340b2945be2102768b307 (table default.Table_9ad8721c3e1e4c549faa983d21798cc2 [id=fc52022e20044cc3bc88ac143e088e69]). Expected version 4 got 3
I20260810 08:00:37.754027 14692 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet 75681d70a3e742768df36a1eaacc3658
I20260810 08:00:37.755805 15278 tablet_service.cc:1514] Processing DeleteTablet for tablet 75681d70a3e742768df36a1eaacc3658 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:37 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:37.757259 15039 heartbeater.cc:347] Connected to a master server at 127.14.57.252:33819
I20260810 08:00:37.769877 15111 tablet_service.cc:1514] Processing DeleteTablet for tablet 75681d70a3e742768df36a1eaacc3658 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:37 UTC) from {username='slave'} at 127.0.0.1:43002
W20260810 08:00:37.772406 14918 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.57.195:37729, user_credentials={real_user=slave}} blocked reactor thread for 56784.9us
I20260810 08:00:37.773711 14918 leader_election.cc:302] T a244ad3bda5d45f9b89f68b734338d21 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224; no voters:
I20260810 08:00:37.776100 15437 raft_consensus.cc:374] T 5990e7ea243843bb869de5dde3e6cd23 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260810 08:00:37.778096 14693 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.14.57.194:55587 (request call id 65) took 848 ms (client timeout 1000 ms). Trace:
W20260810 08:00:37.778563 14693 rpcz_store.cc:269] 0810 08:00:36.929890 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:36.930872 (+ 982us) service_pool.cc:227] Handling call
0810 08:00:37.777998 (+847126us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":2}
I20260810 08:00:37.777402 15437 raft_consensus.cc:729] T 5990e7ea243843bb869de5dde3e6cd23 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:37.789239 15278 tablet_service.cc:1467] Processing CreateTablet for tablet c895f098226642f2adc2f0e3747309b0 (DEFAULT_TABLE table=default.Table_65d76ebe6be54a089add93b91dafcb9e [id=cf3706a7c08346afad4703e3034e8c25]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 08:00:37.793263 14688 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_697df159d8d14451b4f7beeb52f9e54a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:37.794881 15270 tablet_service.cc:1467] Processing CreateTablet for tablet 31c6c0f3b56c4eed8cfab9f91b412e99 (DEFAULT_TABLE table=default.Table_d797c9d1654e4f828ed4b8a7cce2460d [id=3b848cb6be6b40ca979e8daf9bb16591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:37.795840 15437 consensus_queue.cc:260] T 5990e7ea243843bb869de5dde3e6cd23 P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:37.778376 15958 raft_consensus.cc:2781] T a244ad3bda5d45f9b89f68b734338d21 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 08:00:37.797820 14953 tablet_service.cc:1514] Processing DeleteTablet for tablet 75681d70a3e742768df36a1eaacc3658 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:37 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:37.803201 14953 tablet_service.cc:1467] Processing CreateTablet for tablet c895f098226642f2adc2f0e3747309b0 (DEFAULT_TABLE table=default.Table_65d76ebe6be54a089add93b91dafcb9e [id=cf3706a7c08346afad4703e3034e8c25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:37.803478 15988 leader_election.cc:288] T a244ad3bda5d45f9b89f68b734338d21 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Requested vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:37.803506 15958 raft_consensus.cc:686] T a244ad3bda5d45f9b89f68b734338d21 P a91b27762182454baada71e91ceab224 [term 1 LEADER]: Becoming Leader. State: Replica: a91b27762182454baada71e91ceab224, State: Running, Role: LEADER
I20260810 08:00:37.806354 15958 consensus_queue.cc:237] T a244ad3bda5d45f9b89f68b734338d21 P a91b27762182454baada71e91ceab224 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:37.816913 15439 tablet_bootstrap.cc:654] T 8364045e3fa2441e8dc1a1d87f73361d P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:37.818305 14949 tablet_service.cc:1467] Processing CreateTablet for tablet 31c6c0f3b56c4eed8cfab9f91b412e99 (DEFAULT_TABLE table=default.Table_d797c9d1654e4f828ed4b8a7cce2460d [id=3b848cb6be6b40ca979e8daf9bb16591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:37.820720 15278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c895f098226642f2adc2f0e3747309b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:37.822458 14953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c895f098226642f2adc2f0e3747309b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:37.823457 15479 ts_tablet_manager.cc:1918] T 1f427f4093e5417a9ca79b1805b0675e P 9d7219686f304f62a54acc7497cbbd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 08:00:37.823901 15479 log.cc:1196] T 1f427f4093e5417a9ca79b1805b0675e P 9d7219686f304f62a54acc7497cbbd07: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-1/wal/wals/1f427f4093e5417a9ca79b1805b0675e
I20260810 08:00:37.826613 15270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31c6c0f3b56c4eed8cfab9f91b412e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:37.833207 15490 tablet_bootstrap.cc:654] T e1b42eac370145d4b684456639968485 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:37.834153 14949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31c6c0f3b56c4eed8cfab9f91b412e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:37.844135 15479 ts_tablet_manager.cc:1939] T 1f427f4093e5417a9ca79b1805b0675e P 9d7219686f304f62a54acc7497cbbd07: Deleting consensus metadata
I20260810 08:00:37.852159 16002 ts_tablet_manager.cc:1918] T cb37a3a87f8b48a68c19510f306073c2 P 3fc5e568f3724f158b616c303efb9eae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:37.853786 15932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38a7164890624423a7427871aa2b10e0. Latest consensus state: current_term: 2 leader_uuid: "38a7164890624423a7427871aa2b10e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } } }
I20260810 08:00:37.854606 15932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [sys.catalog]: This master's current role is: FOLLOWER
I20260810 08:00:37.858803 15978 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38a7164890624423a7427871aa2b10e0)
I20260810 08:00:37.859654 15978 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:37.865787 16017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "38a7164890624423a7427871aa2b10e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } } }
I20260810 08:00:37.867151 16017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [sys.catalog]: This master's current role is: FOLLOWER
W20260810 08:00:37.877617 16015 log.cc:924] Time spent T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be: Append to log took a long time: real 0.092s user 0.003s sys 0.000s
I20260810 08:00:37.894305 14692 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_7d8f3d372c5c447681edeea50da2a8d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:37.896184 14692 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d8f3d372c5c447681edeea50da2a8d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:37.905395 14690 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet a0887c4efd0f4c7c95e9d12734184a40
I20260810 08:00:37.906986 15800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52382 (request call id 0) took 4746 ms. Trace:
I20260810 08:00:37.907220 14694 master_service.cc:946] ReplaceTablet: received request to replace tablet fcd6d992683c4a5c886d7c5052da76b3 from {username='slave'} at 127.0.0.1:42300
I20260810 08:00:37.909471 14693 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_fcf1860f7d494dddadc19e82024de3c1" } new_table_name: "default.Table_ec141df5dd0d4724a63dd28e7682d9af" modify_external_catalogs: true
I20260810 08:00:37.927063 14956 tablet_service.cc:1514] Processing DeleteTablet for tablet a0887c4efd0f4c7c95e9d12734184a40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:37 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:37.934458 14956 tablet_service.cc:1467] Processing CreateTablet for tablet b9d0908bf8a747afab76f0f36afd7d07 (DEFAULT_TABLE table=default.Table_1dcf09dc44214dae8e1ed6319bdc825f [id=6ab4c475469c438888e3efe0f7a59b3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:37.935683 14956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d0908bf8a747afab76f0f36afd7d07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:37.971781 15437 ts_tablet_manager.cc:1428] T 5990e7ea243843bb869de5dde3e6cd23 P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.372s user 0.013s sys 0.005s
I20260810 08:00:37.988945 14689 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_8caff4077c034aa69c250de0dd3f1a71" } modify_external_catalogs: true
I20260810 08:00:37.989511 14689 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_8caff4077c034aa69c250de0dd3f1a71" } modify_external_catalogs: true
I20260810 08:00:37.881565 15267 tablet_service.cc:1467] Processing CreateTablet for tablet c65ef5216bf04d3caaf83eec9e40e02b (DEFAULT_TABLE table=default.Table_ca56170c42664687b01194771c981e30 [id=321031ecbd1142a69bdef6900e1c15de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:38.008687 15267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c65ef5216bf04d3caaf83eec9e40e02b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:38.008850 14955 tablet_service.cc:1467] Processing CreateTablet for tablet c65ef5216bf04d3caaf83eec9e40e02b (DEFAULT_TABLE table=default.Table_ca56170c42664687b01194771c981e30 [id=321031ecbd1142a69bdef6900e1c15de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:37.887975 15280 tablet_service.cc:1467] Processing CreateTablet for tablet b9d0908bf8a747afab76f0f36afd7d07 (DEFAULT_TABLE table=default.Table_1dcf09dc44214dae8e1ed6319bdc825f [id=6ab4c475469c438888e3efe0f7a59b3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:37.909998 15275 tablet_service.cc:1514] Processing DeleteTablet for tablet a0887c4efd0f4c7c95e9d12734184a40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:37 UTC) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:38.011063 15280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d0908bf8a747afab76f0f36afd7d07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:38.012210 15978 leader_election.cc:288] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 99f2d868d76c440f8443b33e1cd19f58 (127.14.57.254:43757), 38a7164890624423a7427871aa2b10e0 (127.14.57.253:40439)
I20260810 08:00:38.023069 14955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c65ef5216bf04d3caaf83eec9e40e02b. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 08:00:38.028242 14681 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet d25928e4c39d45e69c3efeb36f03aece (table default.Table_a89c79d2a99b415cbac968e3e8a9d53d [id=4c70468b9e8649849cd0e79d59be6bb9], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d25928e4c39d45e69c3efeb36f03aece consensus state has no leader
I20260810 08:00:38.038800 14951 tablet_service.cc:1467] Processing CreateTablet for tablet 78aea80c344a483eb22b9742b8219d61 (DEFAULT_TABLE table=default.Table_4bed023fc139438b8753c1e1cf1dd469 [id=2ab9083d7ffb4485914939c2827547c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:38.041074 14951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78aea80c344a483eb22b9742b8219d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:38.042606 16002 log.cc:1196] T cb37a3a87f8b48a68c19510f306073c2 P 3fc5e568f3724f158b616c303efb9eae: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal/wals/cb37a3a87f8b48a68c19510f306073c2
I20260810 08:00:38.043509 16002 ts_tablet_manager.cc:1939] T cb37a3a87f8b48a68c19510f306073c2 P 3fc5e568f3724f158b616c303efb9eae: Deleting consensus metadata
I20260810 08:00:38.046217 14691 catalog_manager.cc:5495] T a244ad3bda5d45f9b89f68b734338d21 P a91b27762182454baada71e91ceab224 reported cstate change: term changed from 0 to 1, leader changed from <none> to a91b27762182454baada71e91ceab224 (127.14.57.193). New cstate: current_term: 1 leader_uuid: "a91b27762182454baada71e91ceab224" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } health_report { overall_health: UNKNOWN } } }
I20260810 08:00:38.066103 15763 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "a91b27762182454baada71e91ceab224" instance_seqno: 1786348816840822) as {username='slave'} at 127.14.57.193:55435; Asking this server to re-register.
I20260810 08:00:38.067487 15762 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" instance_seqno: 1786348820952802) as {username='slave'} at 127.14.57.195:46585; Asking this server to re-register.
I20260810 08:00:38.068699 15039 heartbeater.cc:464] Registering TS with master...
I20260810 08:00:38.080123 15762 ts_manager.cc:194] Registered new tserver with Master: a91b27762182454baada71e91ceab224 (127.14.57.193:43601)
I20260810 08:00:38.084636 16026 catalog_manager.cc:1259] Loaded cluster ID: eddaf5b0e44343438f86652258a1a0e1
I20260810 08:00:38.087584 15988 raft_consensus.cc:482] T 2d295305b5a843eaa84c8ad666355b84 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fc5e568f3724f158b616c303efb9eae)
I20260810 08:00:38.088120 15988 raft_consensus.cc:504] T 2d295305b5a843eaa84c8ad666355b84 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:38.090788 15988 leader_election.cc:288] T 2d295305b5a843eaa84c8ad666355b84 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
W20260810 08:00:38.093087 14918 leader_election.cc:341] T 2d295305b5a843eaa84c8ad666355b84 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Not found: Tablet not found: 2d295305b5a843eaa84c8ad666355b84
I20260810 08:00:38.115696 14952 tablet_service.cc:1467] Processing CreateTablet for tablet 75c68f15918c4e6d833d10b413b20879 (DEFAULT_TABLE table=default.Table_6099c55f1d194b5f90373405fc5f4977 [id=770fdd87404e42e0831c579626e8071e]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 08:00:38.117086 15079 leader_election.cc:341] T 2fbda1b3b5b4482ba6f03f3e275d94e7 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:38.117561 14952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75c68f15918c4e6d833d10b413b20879. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:38.118201 15079 leader_election.cc:302] T 2fbda1b3b5b4482ba6f03f3e275d94e7 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:38.138357 15145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a244ad3bda5d45f9b89f68b734338d21" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07"
I20260810 08:00:38.139528 15145 raft_consensus.cc:3037] T a244ad3bda5d45f9b89f68b734338d21 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:38.156477 15941 raft_consensus.cc:482] T 655fd33901974b9c85f52f92ff677247 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:38.157205 15941 raft_consensus.cc:504] T 655fd33901974b9c85f52f92ff677247 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:38.161020 15941 leader_election.cc:288] T 655fd33901974b9c85f52f92ff677247 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:38.162508 16026 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be: loading cluster ID for follower catalog manager: success
I20260810 08:00:38.171342 15145 raft_consensus.cc:2454] T a244ad3bda5d45f9b89f68b734338d21 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 1.
I20260810 08:00:38.189607 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e8cadb26fe0428a964f66ef8172f3be" candidate_term: 3 candidate_status { last_received { term: 2 index: 181 } } ignore_live_leader: false dest_uuid: "99f2d868d76c440f8443b33e1cd19f58" is_pre_election: true
I20260810 08:00:38.172199 15980 raft_consensus.cc:2726] T 2fbda1b3b5b4482ba6f03f3e275d94e7 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:38.182401 15490 tablet_bootstrap.cc:492] T e1b42eac370145d4b684456639968485 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:38.192134 14960 tablet_service.cc:1467] Processing CreateTablet for tablet 07606d04e60442e5b6c7407944a26bde (DEFAULT_TABLE table=default.Table_24b98b64df0645a1ac616bc044689f83 [id=8b9cc9ac3f4e402b873febb30b313a48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:38.192924 15490 ts_tablet_manager.cc:1397] T e1b42eac370145d4b684456639968485 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.524s user 0.015s sys 0.002s
I20260810 08:00:38.193437 14960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07606d04e60442e5b6c7407944a26bde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:38.195123 15988 raft_consensus.cc:482] T 5990e7ea243843bb869de5dde3e6cd23 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 08:00:38.195694 15079 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.193:43601, user_credentials={real_user=slave}} blocked reactor thread for 79551.1us
I20260810 08:00:38.195590 15988 raft_consensus.cc:504] T 5990e7ea243843bb869de5dde3e6cd23 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:38.197751 15111 tablet_service.cc:1467] Processing CreateTablet for tablet c895f098226642f2adc2f0e3747309b0 (DEFAULT_TABLE table=default.Table_65d76ebe6be54a089add93b91dafcb9e [id=cf3706a7c08346afad4703e3034e8c25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:38.199118 15111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c895f098226642f2adc2f0e3747309b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:38.203033 15119 tablet_service.cc:1467] Processing CreateTablet for tablet 31c6c0f3b56c4eed8cfab9f91b412e99 (DEFAULT_TABLE table=default.Table_d797c9d1654e4f828ed4b8a7cce2460d [id=3b848cb6be6b40ca979e8daf9bb16591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:38.204232 15119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31c6c0f3b56c4eed8cfab9f91b412e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:38.218923 15490 raft_consensus.cc:348] T e1b42eac370145d4b684456639968485 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:38.219697 15490 raft_consensus.cc:374] T e1b42eac370145d4b684456639968485 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:38.219969 15490 raft_consensus.cc:729] T e1b42eac370145d4b684456639968485 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:38.220578 15490 consensus_queue.cc:260] T e1b42eac370145d4b684456639968485 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:38.223362 15490 ts_tablet_manager.cc:1428] T e1b42eac370145d4b684456639968485 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.030s user 0.002s sys 0.004s
I20260810 08:00:38.225076 15118 tablet_service.cc:1514] Processing DeleteTablet for tablet a0887c4efd0f4c7c95e9d12734184a40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:37 UTC) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:38.225308 15110 tablet_service.cc:1467] Processing CreateTablet for tablet c65ef5216bf04d3caaf83eec9e40e02b (DEFAULT_TABLE table=default.Table_ca56170c42664687b01194771c981e30 [id=321031ecbd1142a69bdef6900e1c15de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:38.227098 15118 tablet_service.cc:1467] Processing CreateTablet for tablet b9d0908bf8a747afab76f0f36afd7d07 (DEFAULT_TABLE table=default.Table_1dcf09dc44214dae8e1ed6319bdc825f [id=6ab4c475469c438888e3efe0f7a59b3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:38.229174 15490 tablet_bootstrap.cc:492] T 33cacf1276d44b24abe6aff12ea8ea51 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:38.228960 15122 tablet_service.cc:1467] Processing CreateTablet for tablet 78aea80c344a483eb22b9742b8219d61 (DEFAULT_TABLE table=default.Table_4bed023fc139438b8753c1e1cf1dd469 [id=2ab9083d7ffb4485914939c2827547c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:38.230273 15437 tablet_bootstrap.cc:492] T e0652a7684b04a89840049ce0d781199 P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:38.230474 15110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c65ef5216bf04d3caaf83eec9e40e02b. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 08:00:38.230223 14573 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.14.57.193:43601, user_credentials={real_user=slave}} blocked reactor thread for 69960.6us
I20260810 08:00:38.234189 15122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78aea80c344a483eb22b9742b8219d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:38.203694 16026 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be: acquiring CA information for follower catalog manager: success
I20260810 08:00:38.238305 14690 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_60fc2701bb3a47e0bb362e24df4ff330" } modify_external_catalogs: true
I20260810 08:00:38.238827 14690 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_60fc2701bb3a47e0bb362e24df4ff330" } modify_external_catalogs: true
I20260810 08:00:38.242509 16026 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 08:00:38.246662 14687 catalog_manager.cc:6040] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: default.Table_9ad8721c3e1e4c549faa983d21798cc2 [id=fc52022e20044cc3bc88ac143e088e69] alter complete (version 4)
I20260810 08:00:38.248148 15118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d0908bf8a747afab76f0f36afd7d07. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 08:00:38.249560 14687 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.14.57.195:51883 (request call id 52) took 1078 ms (client timeout 1000 ms). Trace:
W20260810 08:00:38.252444 14686 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.14.57.193:35941 (request call id 47) took 1317 ms (client timeout 1000 ms). Trace:
W20260810 08:00:38.263864 14686 rpcz_store.cc:269] 0810 08:00:36.934910 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:37.047136 (+112226us) service_pool.cc:227] Handling call
0810 08:00:37.133260 (+ 86124us) spinlock_profiling.cc:243] Waited 54.9 ms on lock 0x7b5400021554. stack: 00007fe2372c4033 00007fe2372c3f30 00007fe236e45534 00000000004db2b4 00000000004db249 00007fe23e1d0af0 00007fe23e253f2a 00007fe24011fd6c 00007fe240219c3e 00007fe23dd50fb4 00007fe23dd50f42 00007fe23dd50e71 00007fe23dd50dec 00007fe23dd500a2 00007fe23bffa94c 00007fe23bff9d86
0810 08:00:38.252378 (+1119118us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":6,"spinlock_wait_cycles":126231936}
I20260810 08:00:38.266173 14686 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_189528c62fa24da3a181b05e3753d54c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:38.267825 14686 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_189528c62fa24da3a181b05e3753d54c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 08:00:38.254041 14687 rpcz_store.cc:269] 0810 08:00:37.171570 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:37.206523 (+ 34953us) service_pool.cc:227] Handling call
0810 08:00:37.439814 (+233291us) spinlock_profiling.cc:243] Waited 231 ms on lock 0x7b5400021554. stack: 00007fe2372c4033 00007fe2372c3f30 00007fe236e45534 00000000004db2b4 00000000004db249 00007fe23e1d0af0 00007fe23e253f2a 00007fe24011fd6c 00007fe240219c3e 00007fe23dd50fb4 00007fe23dd50f42 00007fe23dd50e71 00007fe23dd50dec 00007fe23dd500a2 00007fe23bffa94c 00007fe23bff9d86
0810 08:00:38.249491 (+809677us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 08:00:37.900409 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 08:00:37.900866 (+ 457us) write_op.cc:432] Acquiring schema lock in shared mode
0810 08:00:37.900901 (+ 35us) write_op.cc:435] Acquired schema lock
0810 08:00:37.900914 (+ 13us) tablet.cc:660] Decoding operations
0810 08:00:37.901208 (+ 294us) write_op.cc:620] Acquiring the partition lock for write op
0810 08:00:37.901243 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0810 08:00:37.901262 (+ 19us) tablet.cc:683] Acquiring locks for 1 operations
0810 08:00:37.901409 (+ 147us) tablet.cc:699] Row locks acquired
0810 08:00:37.901422 (+ 13us) write_op.cc:260] PREPARE: finished
0810 08:00:37.901539 (+ 117us) write_op.cc:270] Start()
0810 08:00:37.901661 (+ 122us) write_op.cc:276] Timestamp: P: 1786348837901486 usec, L: 0
0810 08:00:37.901676 (+ 15us) op_driver.cc:331] REPLICATION: starting
0810 08:00:37.986102 (+ 84426us) log.cc:842] Serialized 356 byte log entry
0810 08:00:37.987851 (+ 1749us) spinlock_profiling.cc:243] Waited 83.8 ms on lock 0x7b5400021554. stack: 00007fe2372c4033 00007fe2372c3f30 00007fe236e45534 00000000004db2b4 00000000004db249 00007fe23e1d0471 00007fe23e247198 00007fe23e6a8ade 00007fe23e6a82b8 00007fe23e6acba1 00007fe23e6acb59 00007fe23e6acae9 00007fe23e6acab1 00007fe23e6abdad 0000000000580784 00000000005805b9
0810 08:00:38.241754 (+253903us) op_driver.cc:444] REPLICATION: finished
0810 08:00:38.242797 (+ 1043us) write_op.cc:301] APPLY: starting
0810 08:00:38.242870 (+ 73us) tablet.cc:1365] starting BulkCheckPresence
0810 08:00:38.242971 (+ 101us) tablet.cc:1368] finished BulkCheckPresence
0810 08:00:38.242988 (+ 17us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 08:00:38.243141 (+ 153us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 08:00:38.243267 (+ 126us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 08:00:38.243309 (+ 42us) write_op.cc:312] APPLY: finished
0810 08:00:38.243480 (+ 171us) log.cc:842] Serialized 23 byte log entry
0810 08:00:38.243632 (+ 152us) write_op.cc:489] Releasing partition, row and schema locks
0810 08:00:38.243690 (+ 58us) write_op.cc:454] Released schema lock
0810 08:00:38.243720 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0810 08:00:38.243921 (+ 201us) write_op.cc:489] Releasing partition, row and schema locks
0810 08:00:38.243944 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":6,"spinlock_wait_cycles":531632896,"child_traces":[["op",{"apply.queue_time_us":886,"apply.run_cpu_time_us":1302,"apply.run_wall_time_us":1301,"num_ops":1,"prepare.queue_time_us":132826,"prepare.run_cpu_time_us":2695,"prepare.run_wall_time_us":87582,"replication_time_us":340010,"spinlock_wait_cycles":192841472}]]}
I20260810 08:00:38.272221 14687 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42300:
table { table_name: "default.Table_c65aa418eb904b57aab18f8b403e8c8e" } new_table_name: "default.Table_d734cf52bb724d51802568793b928421" modify_external_catalogs: true
I20260810 08:00:38.280088 14965 tablet_service.cc:1467] Processing CreateTablet for tablet 5615dbf2097d4dcfb2b1e53e64d9bcc3 (DEFAULT_TABLE table=default.Table_7893219bf6754edbba16135d2605adce [id=3afd9e56a6fb4bf5bec58c10b7883f2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:38.280385 14967 tablet_service.cc:1467] Processing CreateTablet for tablet a0887c4efd0f4c7c95e9d12734184a40 (DEFAULT_TABLE table=default.Table_1939fe86b63e4117be863f56bfe7131b [id=9aa3bf24a7844e8e8a56e99dd7ce4c97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:38.280752 15112 tablet_service.cc:1467] Processing CreateTablet for tablet 75c68f15918c4e6d833d10b413b20879 (DEFAULT_TABLE table=default.Table_6099c55f1d194b5f90373405fc5f4977 [id=770fdd87404e42e0831c579626e8071e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:38.281126 14965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5615dbf2097d4dcfb2b1e53e64d9bcc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:38.281757 14963 tablet_service.cc:1467] Processing CreateTablet for tablet fbd50b1476f348589d55d3c43927489d (DEFAULT_TABLE table=default.Table_70ae7b2430d94c3ab90e0d4efc349216 [id=c2b7188fa70c4ecf9fb309c093c870f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:38.282352 15112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75c68f15918c4e6d833d10b413b20879. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:38.282662 14963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbd50b1476f348589d55d3c43927489d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:38.283538 14967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0887c4efd0f4c7c95e9d12734184a40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:38.291057 15113 tablet_service.cc:1467] Processing CreateTablet for tablet 07606d04e60442e5b6c7407944a26bde (DEFAULT_TABLE table=default.Table_24b98b64df0645a1ac616bc044689f83 [id=8b9cc9ac3f4e402b873febb30b313a48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:38.292263 15113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07606d04e60442e5b6c7407944a26bde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:38.300415 15890 ts_tablet_manager.cc:1918] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P a91b27762182454baada71e91ceab224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 08:00:38.300834 15890 log.cc:1196] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P a91b27762182454baada71e91ceab224: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal/wals/cf8bb1c44a7b4d17bddeff8b2fe81b35
W20260810 08:00:38.304379 15234 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.57.193:43601, user_credentials={real_user=slave}} blocked reactor thread for 115401us
I20260810 08:00:38.308197 15490 tablet_bootstrap.cc:654] T 33cacf1276d44b24abe6aff12ea8ea51 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:38.314239 15116 tablet_service.cc:1467] Processing CreateTablet for tablet a8fad494ef944ac5bb571ae4a10fae59 (DEFAULT_TABLE table=default.Table_41b7b5b83e1443748d982b0d63c29acd [id=e9c3d6149fd643949b88f4296acbdd7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:38.315580 15116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8fad494ef944ac5bb571ae4a10fae59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:38.315992 15117 tablet_service.cc:1467] Processing CreateTablet for tablet 2a91dfe4d9fc4edc9ec8c51235a23367 (DEFAULT_TABLE table=default.Table_7f4ac045f2d44f2e9aa549cb790acf5d [id=ec95cdb852f242be8001a7f8beb47625]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:38.317121 15117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a91dfe4d9fc4edc9ec8c51235a23367. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:38.317057 15120 tablet_service.cc:1467] Processing CreateTablet for tablet 5615dbf2097d4dcfb2b1e53e64d9bcc3 (DEFAULT_TABLE table=default.Table_7893219bf6754edbba16135d2605adce [id=3afd9e56a6fb4bf5bec58c10b7883f2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:38.318197 15120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5615dbf2097d4dcfb2b1e53e64d9bcc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:38.318598 15126 tablet_service.cc:1467] Processing CreateTablet for tablet a0887c4efd0f4c7c95e9d12734184a40 (DEFAULT_TABLE table=default.Table_1939fe86b63e4117be863f56bfe7131b [id=9aa3bf24a7844e8e8a56e99dd7ce4c97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:38.319813 15126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0887c4efd0f4c7c95e9d12734184a40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:38.322770 15121 tablet_service.cc:1467] Processing CreateTablet for tablet 6b70393fd05643f4b96b1c99ecd5ba32 (DEFAULT_TABLE table=default.Table_187379f079bf4283824c6b02b86271ba [id=4abdfd680cf74e3791c0389daf6d4e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:38.323801 15121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b70393fd05643f4b96b1c99ecd5ba32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:38.328429 15980 raft_consensus.cc:482] T 0687d323788642f798465ece973702bb P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:38.328884 15980 raft_consensus.cc:504] T 0687d323788642f798465ece973702bb P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:38.315348 15125 tablet_service.cc:1467] Processing CreateTablet for tablet fbd50b1476f348589d55d3c43927489d (DEFAULT_TABLE table=default.Table_70ae7b2430d94c3ab90e0d4efc349216 [id=c2b7188fa70c4ecf9fb309c093c870f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:38.331727 15125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbd50b1476f348589d55d3c43927489d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:38.332084 15980 leader_election.cc:288] T 0687d323788642f798465ece973702bb P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:38.339357 14681 catalog_manager.cc:4841] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): tablet 1f427f4093e5417a9ca79b1805b0675e (table default.Table_e0d4352887e04b38939d453e07a37ef9 [id=11ca1b81b7704d33820eef06f5a25310]) successfully deleted
W20260810 08:00:38.340407 14681 catalog_manager.cc:4821] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): delete failed for tablet 75681d70a3e742768df36a1eaacc3658 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75681d70a3e742768df36a1eaacc3658 already in progress: creating tablet
I20260810 08:00:38.342813 15762 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" instance_seqno: 1786348818967249) as {username='slave'} at 127.14.57.194:51259; Asking this server to re-register.
W20260810 08:00:38.343040 14681 catalog_manager.cc:4808] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): delete failed for tablet a0887c4efd0f4c7c95e9d12734184a40 because the tablet was not found. No further retry: Not found: Tablet not found: a0887c4efd0f4c7c95e9d12734184a40
I20260810 08:00:38.346114 15199 heartbeater.cc:464] Registering TS with master...
I20260810 08:00:38.348100 16040 tablet.cc:1722] T e46648198f5343f7a65079887e5df159 P 3fc5e568f3724f158b616c303efb9eae: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260810 08:00:38.345372 15299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "655fd33901974b9c85f52f92ff677247" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:38.354429 16002 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 189) took 1186 ms. Trace:
I20260810 08:00:38.358201 15762 ts_manager.cc:194] Registered new tserver with Master: 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:38.362282 15988 leader_election.cc:288] T 5990e7ea243843bb869de5dde3e6cd23 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
W20260810 08:00:38.363695 14922 leader_election.cc:341] T 2d295305b5a843eaa84c8ad666355b84 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Not found: Tablet not found: 2d295305b5a843eaa84c8ad666355b84
I20260810 08:00:38.364114 14922 leader_election.cc:302] T 2d295305b5a843eaa84c8ad666355b84 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a91b27762182454baada71e91ceab224; no voters: 3fc5e568f3724f158b616c303efb9eae, 9d7219686f304f62a54acc7497cbbd07
W20260810 08:00:38.390543 15234 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.57.194:36265, user_credentials={real_user=slave}} blocked reactor thread for 54889.7us
I20260810 08:00:38.394018 15270 tablet_service.cc:1467] Processing CreateTablet for tablet 78aea80c344a483eb22b9742b8219d61 (DEFAULT_TABLE table=default.Table_4bed023fc139438b8753c1e1cf1dd469 [id=2ab9083d7ffb4485914939c2827547c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:38.395176 15270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78aea80c344a483eb22b9742b8219d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:38.398830 15278 tablet_service.cc:1467] Processing CreateTablet for tablet 75c68f15918c4e6d833d10b413b20879 (DEFAULT_TABLE table=default.Table_6099c55f1d194b5f90373405fc5f4977 [id=770fdd87404e42e0831c579626e8071e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:38.399847 15278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75c68f15918c4e6d833d10b413b20879. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:38.354869 16002 rpcz_store.cc:276] 0810 08:00:37.167934 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:37.168308 (+ 374us) service_pool.cc:227] Handling call
0810 08:00:37.431821 (+263513us) ts_tablet_manager.cc:1145] Deleting tablet cb37a3a87f8b48a68c19510f306073c2
0810 08:00:37.431858 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:37.778400 (+346542us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:37.851749 (+ 73349us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:38.354364 (+502615us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":56100,"tablet-delete.queue_time_us":262831}
I20260810 08:00:38.402832 16002 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 190) took 1233 ms. Trace:
I20260810 08:00:38.403278 16002 rpcz_store.cc:276] 0810 08:00:37.169448 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:37.169821 (+ 373us) service_pool.cc:227] Handling call
0810 08:00:38.402439 (+1232618us) ts_tablet_manager.cc:1145] Deleting tablet 8b5a629a7f8041179a006fa412fb6b48
0810 08:00:38.402486 (+ 47us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:38.402771 (+ 285us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1231911}
I20260810 08:00:38.404995 16002 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 191) took 1143 ms. Trace:
I20260810 08:00:38.413726 16002 rpcz_store.cc:276] 0810 08:00:37.261683 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:37.262338 (+ 655us) service_pool.cc:227] Handling call
0810 08:00:38.404631 (+1142293us) ts_tablet_manager.cc:1145] Deleting tablet 33cacf1276d44b24abe6aff12ea8ea51
0810 08:00:38.404672 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:38.404942 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1141726}
I20260810 08:00:38.303128 15890 ts_tablet_manager.cc:1939] T cf8bb1c44a7b4d17bddeff8b2fe81b35 P a91b27762182454baada71e91ceab224: Deleting consensus metadata
I20260810 08:00:38.447086 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0687d323788642f798465ece973702bb" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:38.448920 15490 tablet_bootstrap.cc:492] T 33cacf1276d44b24abe6aff12ea8ea51 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:38.449342 15490 ts_tablet_manager.cc:1397] T 33cacf1276d44b24abe6aff12ea8ea51 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.220s user 0.008s sys 0.005s
I20260810 08:00:38.453287 14705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e8cadb26fe0428a964f66ef8172f3be" candidate_term: 3 candidate_status { last_received { term: 2 index: 181 } } ignore_live_leader: false dest_uuid: "38a7164890624423a7427871aa2b10e0" is_pre_election: true
I20260810 08:00:38.455147 15721 leader_election.cc:302] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e8cadb26fe0428a964f66ef8172f3be; no voters: 38a7164890624423a7427871aa2b10e0, 99f2d868d76c440f8443b33e1cd19f58
I20260810 08:00:38.456539 15978 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 08:00:38.424685 15988 raft_consensus.cc:2726] T 2d295305b5a843eaa84c8ad666355b84 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 08:00:38.470384 15490 raft_consensus.cc:348] T 33cacf1276d44b24abe6aff12ea8ea51 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:38.471127 15490 raft_consensus.cc:374] T 33cacf1276d44b24abe6aff12ea8ea51 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:38.471390 15490 raft_consensus.cc:729] T 33cacf1276d44b24abe6aff12ea8ea51 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:38.472093 15490 consensus_queue.cc:260] T 33cacf1276d44b24abe6aff12ea8ea51 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:38.474678 15490 ts_tablet_manager.cc:1428] T 33cacf1276d44b24abe6aff12ea8ea51 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.025s user 0.006s sys 0.001s
I20260810 08:00:38.466053 14957 tablet_service.cc:1467] Processing CreateTablet for tablet 6b70393fd05643f4b96b1c99ecd5ba32 (DEFAULT_TABLE table=default.Table_187379f079bf4283824c6b02b86271ba [id=4abdfd680cf74e3791c0389daf6d4e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:38.467087 16039 tablet.cc:1722] T e46648198f5343f7a65079887e5df159 P a91b27762182454baada71e91ceab224: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260810 08:00:38.474100 14954 tablet_service.cc:1467] Processing CreateTablet for tablet 2a91dfe4d9fc4edc9ec8c51235a23367 (DEFAULT_TABLE table=default.Table_7f4ac045f2d44f2e9aa549cb790acf5d [id=ec95cdb852f242be8001a7f8beb47625]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:38.475029 14949 tablet_service.cc:1467] Processing CreateTablet for tablet a8fad494ef944ac5bb571ae4a10fae59 (DEFAULT_TABLE table=default.Table_41b7b5b83e1443748d982b0d63c29acd [id=e9c3d6149fd643949b88f4296acbdd7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:38.475643 14988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "655fd33901974b9c85f52f92ff677247" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:38.505777 15490 tablet_bootstrap.cc:492] T ae51ddeb22fe4cf7a8aa2d8b2b70a535 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:38.506302 15980 raft_consensus.cc:482] T e1b42eac370145d4b684456639968485 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:38.506719 15980 raft_consensus.cc:504] T e1b42eac370145d4b684456639968485 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:38.508630 15980 leader_election.cc:288] T e1b42eac370145d4b684456639968485 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:38.550195 14957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b70393fd05643f4b96b1c99ecd5ba32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:38.554925 14988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b42eac370145d4b684456639968485" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:38.571197 14949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8fad494ef944ac5bb571ae4a10fae59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:38.578732 14954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a91dfe4d9fc4edc9ec8c51235a23367. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:38.587967 15890 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34630 (request call id 147) took 4735 ms. Trace:
I20260810 08:00:38.588618 15890 rpcz_store.cc:276] 0810 08:00:33.852308 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:33.852494 (+ 186us) service_pool.cc:227] Handling call
0810 08:00:37.425664 (+3573170us) ts_tablet_manager.cc:1145] Deleting tablet cf8bb1c44a7b4d17bddeff8b2fe81b35
0810 08:00:37.425761 (+ 97us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:38.011028 (+585267us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:38.300163 (+289135us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:38.587821 (+287658us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3572471}
I20260810 08:00:38.591298 15890 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34630 (request call id 148) took 4737 ms. Trace:
I20260810 08:00:38.543964 15490 tablet_bootstrap.cc:654] T ae51ddeb22fe4cf7a8aa2d8b2b70a535 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
W20260810 08:00:38.537598 15079 leader_election.cc:341] T 0687d323788642f798465ece973702bb P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
W20260810 08:00:38.621119 15079 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.193:43601, user_credentials={real_user=slave}} blocked reactor thread for 83567.1us
W20260810 08:00:38.621765 15079 leader_election.cc:341] T 655fd33901974b9c85f52f92ff677247 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
W20260810 08:00:38.622398 15079 leader_election.cc:341] T e1b42eac370145d4b684456639968485 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
W20260810 08:00:38.626911 15076 leader_election.cc:341] T 655fd33901974b9c85f52f92ff677247 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:38.627543 15076 leader_election.cc:302] T 655fd33901974b9c85f52f92ff677247 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:38.630208 15980 raft_consensus.cc:2726] T 655fd33901974b9c85f52f92ff677247 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:38.636757 14678 catalog_manager.cc:4841] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): tablet cb37a3a87f8b48a68c19510f306073c2 (table default.Table_64e1c230e9624fc9ba6c328346150ffe [id=46e6c2115d8c4b5e9907ed76b273638a]) successfully deleted
W20260810 08:00:38.637404 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet 8b5a629a7f8041179a006fa412fb6b48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8b5a629a7f8041179a006fa412fb6b48 already in progress: creating tablet
I20260810 08:00:38.423343 15275 tablet_service.cc:1467] Processing CreateTablet for tablet 07606d04e60442e5b6c7407944a26bde (DEFAULT_TABLE table=default.Table_24b98b64df0645a1ac616bc044689f83 [id=8b9cc9ac3f4e402b873febb30b313a48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:38.426136 15268 tablet_service.cc:1467] Processing CreateTablet for tablet a8fad494ef944ac5bb571ae4a10fae59 (DEFAULT_TABLE table=default.Table_41b7b5b83e1443748d982b0d63c29acd [id=e9c3d6149fd643949b88f4296acbdd7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:38.642391 15275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07606d04e60442e5b6c7407944a26bde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:38.633949 15890 rpcz_store.cc:276] 0810 08:00:33.853807 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:33.853928 (+ 121us) service_pool.cc:227] Handling call
0810 08:00:38.590592 (+4736664us) ts_tablet_manager.cc:1145] Deleting tablet d3b41c4912f840c18913fe29c1af9fd9
0810 08:00:38.590718 (+ 126us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:38.591240 (+ 522us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4736116}
I20260810 08:00:38.427625 15273 tablet_service.cc:1467] Processing CreateTablet for tablet 2a91dfe4d9fc4edc9ec8c51235a23367 (DEFAULT_TABLE table=default.Table_7f4ac045f2d44f2e9aa549cb790acf5d [id=ec95cdb852f242be8001a7f8beb47625]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:38.643141 15268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8fad494ef944ac5bb571ae4a10fae59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:38.643955 15890 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34630 (request call id 153) took 4109 ms. Trace:
I20260810 08:00:38.644433 15273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a91dfe4d9fc4edc9ec8c51235a23367. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:38.429040 15271 tablet_service.cc:1467] Processing CreateTablet for tablet a0887c4efd0f4c7c95e9d12734184a40 (DEFAULT_TABLE table=default.Table_1939fe86b63e4117be863f56bfe7131b [id=9aa3bf24a7844e8e8a56e99dd7ce4c97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:38.648200 15271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0887c4efd0f4c7c95e9d12734184a40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:38.620303 15439 tablet_bootstrap.cc:492] T 8364045e3fa2441e8dc1a1d87f73361d P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:38.630424 15276 tablet_service.cc:1467] Processing CreateTablet for tablet fbd50b1476f348589d55d3c43927489d (DEFAULT_TABLE table=default.Table_70ae7b2430d94c3ab90e0d4efc349216 [id=c2b7188fa70c4ecf9fb309c093c870f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:38.632294 15269 tablet_service.cc:1467] Processing CreateTablet for tablet 5615dbf2097d4dcfb2b1e53e64d9bcc3 (DEFAULT_TABLE table=default.Table_7893219bf6754edbba16135d2605adce [id=3afd9e56a6fb4bf5bec58c10b7883f2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:38.655274 15276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbd50b1476f348589d55d3c43927489d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:38.654997 15980 raft_consensus.cc:482] T 33cacf1276d44b24abe6aff12ea8ea51 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:38.656038 15980 raft_consensus.cc:504] T 33cacf1276d44b24abe6aff12ea8ea51 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:38.658497 15269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5615dbf2097d4dcfb2b1e53e64d9bcc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:38.659978 15980 leader_election.cc:288] T 33cacf1276d44b24abe6aff12ea8ea51 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:38.635758 15354 heartbeater.cc:464] Registering TS with master...
I20260810 08:00:38.638084 15277 tablet_service.cc:1467] Processing CreateTablet for tablet 6b70393fd05643f4b96b1c99ecd5ba32 (DEFAULT_TABLE table=default.Table_187379f079bf4283824c6b02b86271ba [id=4abdfd680cf74e3791c0389daf6d4e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 08:00:38.666320 14988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33cacf1276d44b24abe6aff12ea8ea51" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:38.666951 15277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b70393fd05643f4b96b1c99ecd5ba32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 08:00:38.644414 15890 rpcz_store.cc:276] 0810 08:00:34.534487 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:34.534663 (+ 176us) service_pool.cc:227] Handling call
0810 08:00:38.643187 (+4108524us) ts_tablet_manager.cc:1145] Deleting tablet 579bac622ddf40bf80d01d66ae2fa100
0810 08:00:38.643342 (+ 155us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:38.643881 (+ 539us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4107874}
I20260810 08:00:38.671713 15890 tablet_replica.cc:330] T 2d295305b5a843eaa84c8ad666355b84 P a91b27762182454baada71e91ceab224: stopping tablet replica
I20260810 08:00:38.673174 15890 raft_consensus.cc:2229] T 2d295305b5a843eaa84c8ad666355b84 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:38.676225 15490 tablet_bootstrap.cc:492] T ae51ddeb22fe4cf7a8aa2d8b2b70a535 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:38.676806 15490 ts_tablet_manager.cc:1397] T ae51ddeb22fe4cf7a8aa2d8b2b70a535 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.171s user 0.016s sys 0.000s
I20260810 08:00:38.678285 16002 tablet_replica.cc:330] T 1f427f4093e5417a9ca79b1805b0675e P 3fc5e568f3724f158b616c303efb9eae: stopping tablet replica
I20260810 08:00:38.679548 16002 raft_consensus.cc:2229] T 1f427f4093e5417a9ca79b1805b0675e P 3fc5e568f3724f158b616c303efb9eae [term 1 LEADER]: Raft consensus shutting down.
I20260810 08:00:38.680804 16002 raft_consensus.cc:2258] T 1f427f4093e5417a9ca79b1805b0675e P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:38.680990 15490 raft_consensus.cc:348] T ae51ddeb22fe4cf7a8aa2d8b2b70a535 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:38.682001 15490 raft_consensus.cc:374] T ae51ddeb22fe4cf7a8aa2d8b2b70a535 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:38.683667 15437 tablet_bootstrap.cc:654] T e0652a7684b04a89840049ce0d781199 P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
W20260810 08:00:38.685468 15079 leader_election.cc:341] T 33cacf1276d44b24abe6aff12ea8ea51 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:38.682268 15490 raft_consensus.cc:729] T ae51ddeb22fe4cf7a8aa2d8b2b70a535 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:38.693825 15490 consensus_queue.cc:260] T ae51ddeb22fe4cf7a8aa2d8b2b70a535 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:38.697906 15439 ts_tablet_manager.cc:1397] T 8364045e3fa2441e8dc1a1d87f73361d P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 1.024s user 0.069s sys 0.065s
I20260810 08:00:38.704671 15439 raft_consensus.cc:348] T 8364045e3fa2441e8dc1a1d87f73361d P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:38.705093 15890 pending_rounds.cc:62] T 2d295305b5a843eaa84c8ad666355b84 P a91b27762182454baada71e91ceab224: Trying to abort 1 pending ops.
I20260810 08:00:38.706346 15890 pending_rounds.cc:68] T 2d295305b5a843eaa84c8ad666355b84 P a91b27762182454baada71e91ceab224: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7316884827538821120 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "2d295305b5a843eaa84c8ad666355b84" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.Table_d5c08115c8b24ad48a9686e48a45ab82" dest_uuid: "3fc5e568f3724f158b616c303efb9eae" new_extra_config { } }
I20260810 08:00:38.712265 15890 raft_consensus.cc:2258] T 2d295305b5a843eaa84c8ad666355b84 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 08:00:38.726998 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet 33cacf1276d44b24abe6aff12ea8ea51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33cacf1276d44b24abe6aff12ea8ea51 already in progress: creating tablet
I20260810 08:00:38.735679 15762 ts_manager.cc:194] Registered new tserver with Master: 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:38.747100 15490 ts_tablet_manager.cc:1428] T ae51ddeb22fe4cf7a8aa2d8b2b70a535 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.070s user 0.007s sys 0.001s
W20260810 08:00:38.758589 14922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.194:36265, user_credentials={real_user=slave}} blocked reactor thread for 394886us
I20260810 08:00:38.760600 15890 ts_tablet_manager.cc:1905] T 2d295305b5a843eaa84c8ad666355b84 P a91b27762182454baada71e91ceab224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:38.763769 14681 catalog_manager.cc:4841] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): tablet cf8bb1c44a7b4d17bddeff8b2fe81b35 (table default.Table_a6ce85a21a354e7c92f158c1ea3212b0 [id=8329ae05a9b0470897f5830376c1f87a]) successfully deleted
I20260810 08:00:38.764789 15490 tablet_bootstrap.cc:492] T a92d01d83dbf4b6ba47464dcefa8b2ae P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
W20260810 08:00:38.776273 14681 catalog_manager.cc:4821] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet d3b41c4912f840c18913fe29c1af9fd9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d3b41c4912f840c18913fe29c1af9fd9 already in progress: creating tablet
W20260810 08:00:38.809082 14681 catalog_manager.cc:4821] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet 579bac622ddf40bf80d01d66ae2fa100 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 579bac622ddf40bf80d01d66ae2fa100 already in progress: creating tablet
I20260810 08:00:38.825881 15958 consensus_queue.cc:1035] T a244ad3bda5d45f9b89f68b734338d21 P a91b27762182454baada71e91ceab224 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260810 08:00:38.831912 15076 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.57.195:37729, user_credentials={real_user=slave}} blocked reactor thread for 97025.6us
I20260810 08:00:38.838613 15437 tablet_bootstrap.cc:492] T e0652a7684b04a89840049ce0d781199 P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:38.839079 15437 ts_tablet_manager.cc:1397] T e0652a7684b04a89840049ce0d781199 P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.866s user 0.007s sys 0.015s
I20260810 08:00:38.726282 15299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0687d323788642f798465ece973702bb" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:38.729997 15300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33cacf1276d44b24abe6aff12ea8ea51" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:38.731781 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b42eac370145d4b684456639968485" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:38.844313 15980 raft_consensus.cc:482] T 389b30cba2b149e9ab0de8c191808db2 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:38.743602 16002 ts_tablet_manager.cc:1905] T 1f427f4093e5417a9ca79b1805b0675e P 3fc5e568f3724f158b616c303efb9eae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:38.796133 15302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5990e7ea243843bb869de5dde3e6cd23" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:38.845106 15980 raft_consensus.cc:504] T 389b30cba2b149e9ab0de8c191808db2 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:38.846940 15437 raft_consensus.cc:348] T e0652a7684b04a89840049ce0d781199 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
W20260810 08:00:38.847590 14918 leader_election.cc:341] T 5990e7ea243843bb869de5dde3e6cd23 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:38.847769 15437 raft_consensus.cc:374] T e0652a7684b04a89840049ce0d781199 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:38.848110 15437 raft_consensus.cc:729] T e0652a7684b04a89840049ce0d781199 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
W20260810 08:00:38.849049 15076 leader_election.cc:341] T 0687d323788642f798465ece973702bb P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:38.849767 15439 raft_consensus.cc:374] T 8364045e3fa2441e8dc1a1d87f73361d P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:38.850195 15076 leader_election.cc:302] T 0687d323788642f798465ece973702bb P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:38.851219 15437 consensus_queue.cc:260] T e0652a7684b04a89840049ce0d781199 P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:38.854292 16050 raft_consensus.cc:2726] T 0687d323788642f798465ece973702bb P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:38.855288 15490 tablet_bootstrap.cc:654] T a92d01d83dbf4b6ba47464dcefa8b2ae P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
W20260810 08:00:38.857903 15076 leader_election.cc:341] T 33cacf1276d44b24abe6aff12ea8ea51 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:38.858755 15076 leader_election.cc:302] T 33cacf1276d44b24abe6aff12ea8ea51 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
W20260810 08:00:38.860582 15076 leader_election.cc:341] T e1b42eac370145d4b684456639968485 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:38.861311 16050 raft_consensus.cc:2726] T 33cacf1276d44b24abe6aff12ea8ea51 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:38.862380 15437 ts_tablet_manager.cc:1428] T e0652a7684b04a89840049ce0d781199 P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.023s user 0.008s sys 0.000s
W20260810 08:00:38.862316 14922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.57.194:36265, user_credentials={real_user=slave}} blocked reactor thread for 51022.7us
I20260810 08:00:38.863459 15437 tablet_bootstrap.cc:492] T 389b30cba2b149e9ab0de8c191808db2 P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:38.863412 15439 raft_consensus.cc:729] T 8364045e3fa2441e8dc1a1d87f73361d P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:38.873237 14988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "389b30cba2b149e9ab0de8c191808db2" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
W20260810 08:00:38.874277 14922 leader_election.cc:341] T 5990e7ea243843bb869de5dde3e6cd23 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Not found: Tablet not found: 5990e7ea243843bb869de5dde3e6cd23
I20260810 08:00:38.874734 14922 leader_election.cc:302] T 5990e7ea243843bb869de5dde3e6cd23 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a91b27762182454baada71e91ceab224; no voters: 3fc5e568f3724f158b616c303efb9eae, 9d7219686f304f62a54acc7497cbbd07
W20260810 08:00:38.875542 15079 leader_election.cc:341] T 389b30cba2b149e9ab0de8c191808db2 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:38.878683 15980 leader_election.cc:288] T 389b30cba2b149e9ab0de8c191808db2 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:38.869599 15439 consensus_queue.cc:260] T 8364045e3fa2441e8dc1a1d87f73361d P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:38.890309 16054 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38a7164890624423a7427871aa2b10e0)
I20260810 08:00:38.890887 15076 leader_election.cc:302] T e1b42eac370145d4b684456639968485 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:38.894377 15980 raft_consensus.cc:2726] T e1b42eac370145d4b684456639968485 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:38.895529 16023 raft_consensus.cc:2726] T 5990e7ea243843bb869de5dde3e6cd23 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:38.899592 16023 consensus_queue.cc:1035] T a244ad3bda5d45f9b89f68b734338d21 P a91b27762182454baada71e91ceab224 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 08:00:38.900907 16054 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:38.917402 15490 tablet_bootstrap.cc:492] T a92d01d83dbf4b6ba47464dcefa8b2ae P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:38.917789 15437 tablet_bootstrap.cc:654] T 389b30cba2b149e9ab0de8c191808db2 P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:38.917964 15490 ts_tablet_manager.cc:1397] T a92d01d83dbf4b6ba47464dcefa8b2ae P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.164s user 0.003s sys 0.014s
I20260810 08:00:38.937295 15439 ts_tablet_manager.cc:1428] T 8364045e3fa2441e8dc1a1d87f73361d P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.239s user 0.006s sys 0.001s
I20260810 08:00:38.938887 15280 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40692 (request call id 197) took 1051 ms. Trace:
I20260810 08:00:38.939188 15490 raft_consensus.cc:348] T a92d01d83dbf4b6ba47464dcefa8b2ae P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:38.945757 15302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "389b30cba2b149e9ab0de8c191808db2" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
W20260810 08:00:38.935164 15267 consensus_meta.cc:287] T c65ef5216bf04d3caaf83eec9e40e02b P 3fc5e568f3724f158b616c303efb9eae: Time spent flushing consensus metadata: real 0.516s user 0.009s sys 0.001s
I20260810 08:00:38.950074 14705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99f2d868d76c440f8443b33e1cd19f58" candidate_term: 3 candidate_status { last_received { term: 2 index: 280 } } ignore_live_leader: false dest_uuid: "38a7164890624423a7427871aa2b10e0" is_pre_election: true
I20260810 08:00:38.960965 16043 raft_consensus.cc:482] T 8364045e3fa2441e8dc1a1d87f73361d P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:38.962807 15267 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40692 (request call id 196) took 1083 ms. Trace:
I20260810 08:00:38.963176 15267 rpcz_store.cc:276] 0810 08:00:37.879574 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:37.880238 (+ 664us) service_pool.cc:227] Handling call
0810 08:00:38.008145 (+127907us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 08:00:38.008257 (+ 112us) ts_tablet_manager.cc:676] Creating new metadata...
0810 08:00:38.418665 (+410408us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:38.962291 (+543626us) tablet_replica.cc:205] Creating consensus instance
0810 08:00:38.962729 (+ 438us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 08:00:38.961434 16043 raft_consensus.cc:504] T 8364045e3fa2441e8dc1a1d87f73361d P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:38.976037 16043 leader_election.cc:288] T 8364045e3fa2441e8dc1a1d87f73361d P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:38.981725 16054 leader_election.cc:288] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 38a7164890624423a7427871aa2b10e0 (127.14.57.253:40439), 1e8cadb26fe0428a964f66ef8172f3be (127.14.57.252:33819)
I20260810 08:00:38.940194 15490 raft_consensus.cc:374] T a92d01d83dbf4b6ba47464dcefa8b2ae P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260810 08:00:38.960295 15076 leader_election.cc:341] T 389b30cba2b149e9ab0de8c191808db2 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:38.983494 15490 raft_consensus.cc:729] T a92d01d83dbf4b6ba47464dcefa8b2ae P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:38.983816 15076 leader_election.cc:302] T 389b30cba2b149e9ab0de8c191808db2 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:38.984390 15490 consensus_queue.cc:260] T a92d01d83dbf4b6ba47464dcefa8b2ae P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:38.985775 15980 raft_consensus.cc:2726] T 389b30cba2b149e9ab0de8c191808db2 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:38.986819 15490 ts_tablet_manager.cc:1428] T a92d01d83dbf4b6ba47464dcefa8b2ae P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.063s user 0.004s sys 0.003s
I20260810 08:00:38.987776 15490 tablet_bootstrap.cc:492] T 461de46896ee492b9f7cd1202208c19a P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
W20260810 08:00:38.977445 15234 leader_election.cc:341] T 8364045e3fa2441e8dc1a1d87f73361d P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Not found: Tablet not found: 8364045e3fa2441e8dc1a1d87f73361d
I20260810 08:00:39.016278 16065 tablet.cc:1722] T e46648198f5343f7a65079887e5df159 P 9d7219686f304f62a54acc7497cbbd07: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20260810 08:00:39.020064 15270 consensus_meta.cc:287] T 78aea80c344a483eb22b9742b8219d61 P 3fc5e568f3724f158b616c303efb9eae: Time spent flushing consensus metadata: real 0.577s user 0.007s sys 0.001s
I20260810 08:00:39.040987 15490 tablet_bootstrap.cc:654] T 461de46896ee492b9f7cd1202208c19a P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:38.939525 15280 rpcz_store.cc:276] 0810 08:00:37.887166 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:37.887275 (+ 109us) service_pool.cc:227] Handling call
0810 08:00:38.009896 (+122621us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 08:00:38.009987 (+ 91us) ts_tablet_manager.cc:676] Creating new metadata...
0810 08:00:38.611339 (+601352us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:38.936183 (+324844us) tablet_replica.cc:205] Creating consensus instance
0810 08:00:38.938830 (+ 2647us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 08:00:39.026419 15439 tablet_bootstrap.cc:492] T 579bac622ddf40bf80d01d66ae2fa100 P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:39.057024 15797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99f2d868d76c440f8443b33e1cd19f58" candidate_term: 3 candidate_status { last_received { term: 2 index: 280 } } ignore_live_leader: false dest_uuid: "1e8cadb26fe0428a964f66ef8172f3be" is_pre_election: true
I20260810 08:00:39.057924 15797 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99f2d868d76c440f8443b33e1cd19f58 in term 2.
I20260810 08:00:39.059391 14598 leader_election.cc:302] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e8cadb26fe0428a964f66ef8172f3be, 99f2d868d76c440f8443b33e1cd19f58; no voters: 38a7164890624423a7427871aa2b10e0
I20260810 08:00:39.067535 16054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 08:00:39.071125 16054 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 38a7164890624423a7427871aa2b10e0)
I20260810 08:00:39.071604 16054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 2 FOLLOWER]: Advancing to term 3
W20260810 08:00:39.078612 14922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.57.194:36265, user_credentials={real_user=slave}} blocked reactor thread for 47585.7us
I20260810 08:00:39.088948 15490 tablet_bootstrap.cc:492] T 461de46896ee492b9f7cd1202208c19a P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:39.089337 15490 ts_tablet_manager.cc:1397] T 461de46896ee492b9f7cd1202208c19a P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.102s user 0.000s sys 0.009s
I20260810 08:00:39.089175 14988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8364045e3fa2441e8dc1a1d87f73361d" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:39.089810 14988 raft_consensus.cc:2454] T 8364045e3fa2441e8dc1a1d87f73361d P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 0.
I20260810 08:00:39.091662 15490 raft_consensus.cc:348] T 461de46896ee492b9f7cd1202208c19a P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:39.092324 15490 raft_consensus.cc:374] T 461de46896ee492b9f7cd1202208c19a P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:39.092567 15490 raft_consensus.cc:729] T 461de46896ee492b9f7cd1202208c19a P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:39.093257 15490 consensus_queue.cc:260] T 461de46896ee492b9f7cd1202208c19a P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:39.095242 15490 ts_tablet_manager.cc:1428] T 461de46896ee492b9f7cd1202208c19a P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260810 08:00:39.103194 14693 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8980090fc4ff4de68bbb690f30c3f46e consensus state has no leader
I20260810 08:00:39.103689 14693 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:42300 (request call id 254) took 1240 ms. Trace:
I20260810 08:00:39.103962 14693 rpcz_store.cc:276] 0810 08:00:37.863799 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:37.863990 (+ 191us) service_pool.cc:227] Handling call
0810 08:00:37.909380 (+ 45390us) spinlock_profiling.cc:243] Waited 44.7 ms on lock 0x7b5400021554. stack: 00007fe2372c4033 00007fe2372c3f30 00007fe236e45534 00000000004db2b4 00000000004db249 00007fe23e1d0af0 00007fe23e253f2a 00007fe24011fd6c 00007fe24021b7d6 00007fe23dd61634 00007fe23dd615c2 00007fe23dd614f1 00007fe23dd6146c 00007fe23dd60722 00007fe23bffa94c 00007fe23bff9d86
0810 08:00:37.910100 (+ 720us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 08:00:37.910710 (+ 610us) catalog_manager.cc:3616] Apply alter schema
0810 08:00:37.911234 (+ 524us) catalog_manager.cc:3639] Acquired catalog manager lock
0810 08:00:37.911413 (+ 179us) catalog_manager.cc:3786] Updating metadata on disk
0810 08:00:39.102646 (+1191233us) catalog_manager.cc:3817] Committing alterations to in-memory state
0810 08:00:39.103637 (+ 991us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 08:00:38.292805 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 08:00:38.293117 (+ 312us) write_op.cc:432] Acquiring schema lock in shared mode
0810 08:00:38.293140 (+ 23us) write_op.cc:435] Acquired schema lock
0810 08:00:38.293151 (+ 11us) tablet.cc:660] Decoding operations
0810 08:00:38.293375 (+ 224us) write_op.cc:620] Acquiring the partition lock for write op
0810 08:00:38.293407 (+ 32us) write_op.cc:641] Partition lock acquired for write op
0810 08:00:38.293427 (+ 20us) tablet.cc:683] Acquiring locks for 1 operations
0810 08:00:38.293550 (+ 123us) tablet.cc:699] Row locks acquired
0810 08:00:38.293563 (+ 13us) write_op.cc:260] PREPARE: finished
0810 08:00:38.329362 (+ 35799us) write_op.cc:270] Start()
0810 08:00:38.329467 (+ 105us) write_op.cc:276] Timestamp: P: 1786348838329298 usec, L: 0
0810 08:00:38.329486 (+ 19us) op_driver.cc:331] REPLICATION: starting
0810 08:00:38.330148 (+ 662us) log.cc:842] Serialized 405 byte log entry
0810 08:00:39.074374 (+744226us) op_driver.cc:444] REPLICATION: finished
0810 08:00:39.083864 (+ 9490us) write_op.cc:301] APPLY: starting
0810 08:00:39.083923 (+ 59us) tablet.cc:1365] starting BulkCheckPresence
0810 08:00:39.084013 (+ 90us) tablet.cc:1368] finished BulkCheckPresence
0810 08:00:39.084141 (+ 128us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 08:00:39.101805 (+ 17664us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 08:00:39.101949 (+ 144us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 08:00:39.101996 (+ 47us) write_op.cc:312] APPLY: finished
0810 08:00:39.102190 (+ 194us) log.cc:842] Serialized 23 byte log entry
0810 08:00:39.102348 (+ 158us) write_op.cc:489] Releasing partition, row and schema locks
0810 08:00:39.102404 (+ 56us) write_op.cc:454] Released schema lock
0810 08:00:39.102434 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":102713728,"child_traces":[["op",{"apply.queue_time_us":9293,"num_ops":1,"prepare.queue_time_us":380244,"prepare.run_cpu_time_us":1978,"prepare.run_wall_time_us":39582,"replication_time_us":744803,"threads_started":1}]]}
I20260810 08:00:39.123703 14689 catalog_manager.cc:5777] Not sending DeleteTablet requests; no consensus state for tablet a6c580c4ad7e4e13b7531c66cb2616fe
I20260810 08:00:39.124046 14689 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:42300 (request call id 257) took 1135 ms. Trace:
I20260810 08:00:39.124295 14689 rpcz_store.cc:276] 0810 08:00:37.988525 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:37.988628 (+ 103us) service_pool.cc:227] Handling call
0810 08:00:37.989935 (+ 1307us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 08:00:37.991031 (+ 1096us) catalog_manager.cc:2752] Modifying in-memory table state
0810 08:00:37.991157 (+ 126us) catalog_manager.cc:2761] Locking tablets
0810 08:00:37.991630 (+ 473us) catalog_manager.cc:2775] Removing table and tablets from system table
0810 08:00:39.123330 (+1131700us) catalog_manager.cc:2793] Removing table from by-name map
0810 08:00:39.123516 (+ 186us) catalog_manager.cc:2809] Committing in-memory state
0810 08:00:39.123623 (+ 107us) catalog_manager.cc:2813] Aborting table tasks
0810 08:00:39.123984 (+ 361us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 08:00:38.333214 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 08:00:38.333606 (+ 392us) write_op.cc:432] Acquiring schema lock in shared mode
0810 08:00:38.333654 (+ 48us) write_op.cc:435] Acquired schema lock
0810 08:00:38.333667 (+ 13us) tablet.cc:660] Decoding operations
0810 08:00:38.333958 (+ 291us) write_op.cc:620] Acquiring the partition lock for write op
0810 08:00:38.333994 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0810 08:00:38.334016 (+ 22us) tablet.cc:683] Acquiring locks for 2 operations
0810 08:00:38.334201 (+ 185us) tablet.cc:699] Row locks acquired
0810 08:00:38.334214 (+ 13us) write_op.cc:260] PREPARE: finished
0810 08:00:38.334309 (+ 95us) write_op.cc:270] Start()
0810 08:00:38.334399 (+ 90us) write_op.cc:276] Timestamp: P: 1786348838334273 usec, L: 0
0810 08:00:38.334414 (+ 15us) op_driver.cc:331] REPLICATION: starting
0810 08:00:38.710481 (+376067us) log.cc:842] Serialized 533 byte log entry
0810 08:00:39.120107 (+409626us) op_driver.cc:444] REPLICATION: finished
0810 08:00:39.121188 (+ 1081us) write_op.cc:301] APPLY: starting
0810 08:00:39.121255 (+ 67us) tablet.cc:1365] starting BulkCheckPresence
0810 08:00:39.121365 (+ 110us) tablet.cc:1368] finished BulkCheckPresence
0810 08:00:39.121499 (+ 134us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 08:00:39.121848 (+ 349us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 08:00:39.122005 (+ 157us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0810 08:00:39.122073 (+ 68us) write_op.cc:312] APPLY: finished
0810 08:00:39.122269 (+ 196us) log.cc:842] Serialized 29 byte log entry
0810 08:00:39.122918 (+ 649us) write_op.cc:489] Releasing partition, row and schema locks
0810 08:00:39.122977 (+ 59us) write_op.cc:454] Released schema lock
0810 08:00:39.123009 (+ 32us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":905,"num_ops":2,"prepare.queue_time_us":339370,"prepare.run_cpu_time_us":2461,"prepare.run_wall_time_us":378665,"replication_time_us":785623,"spinlock_wait_cycles":726400}]]}
I20260810 08:00:39.134626 15490 tablet_bootstrap.cc:492] T fd97cbfb16bf4ee2b4eed1e8bbbc7224 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:39.145932 14695 catalog_manager.cc:5587] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265) does not have the latest schema for tablet a4a78b2a960d4d78aa1f48a9c9edf3be (table default.Table_89fe7ad723964551909b73ff6b72b9af [id=ce96e7fb7c46404a859ca08c1fd1b26a]). Expected version 1 got 0
I20260810 08:00:39.146831 14695 catalog_manager.cc:5587] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265) does not have the latest schema for tablet 0687d323788642f798465ece973702bb (table default.Table_c65aa418eb904b57aab18f8b403e8c8e [id=0d60b763c4804818ae26fa693a4fd808]). Expected version 1 got 0
I20260810 08:00:39.158476 14693 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_79deedd2ebe345a9ac1f5da91c94595a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:39.159982 14693 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79deedd2ebe345a9ac1f5da91c94595a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:39.160380 14690 catalog_manager.cc:5777] Not sending DeleteTablet requests; no consensus state for tablet 12cb9de6c985455e85276a5f7501906d
I20260810 08:00:39.160699 14690 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:42300 (request call id 258) took 1047 ms. Trace:
I20260810 08:00:39.163066 15490 tablet_bootstrap.cc:654] T fd97cbfb16bf4ee2b4eed1e8bbbc7224 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:39.160972 14690 rpcz_store.cc:276] 0810 08:00:38.113677 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:38.113802 (+ 125us) service_pool.cc:227] Handling call
0810 08:00:38.238216 (+124414us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b5400021554. stack: 00007fe2372c4033 00007fe2372c3f30 00007fe236e45534 00000000004db2b4 00000000004db249 00007fe23e1d0af0 00007fe23e253f2a 00007fe24011fd6c 00007fe24021b466 00007fe23dd5cb34 00007fe23dd5cac2 00007fe23dd5c9f1 00007fe23dd5c96c 00007fe23dd5bc22 00007fe23bffa94c 00007fe23bff9d86
0810 08:00:38.239196 (+ 980us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 08:00:38.239674 (+ 478us) catalog_manager.cc:2752] Modifying in-memory table state
0810 08:00:38.239789 (+ 115us) catalog_manager.cc:2761] Locking tablets
0810 08:00:38.240243 (+ 454us) catalog_manager.cc:2775] Removing table and tablets from system table
0810 08:00:39.160031 (+919788us) catalog_manager.cc:2793] Removing table from by-name map
0810 08:00:39.160212 (+ 181us) catalog_manager.cc:2809] Committing in-memory state
0810 08:00:39.160313 (+ 101us) catalog_manager.cc:2813] Aborting table tasks
0810 08:00:39.160650 (+ 337us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 08:00:38.805988 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 08:00:38.857428 (+ 51440us) write_op.cc:432] Acquiring schema lock in shared mode
0810 08:00:38.857452 (+ 24us) write_op.cc:435] Acquired schema lock
0810 08:00:38.857465 (+ 13us) tablet.cc:660] Decoding operations
0810 08:00:38.857860 (+ 395us) write_op.cc:620] Acquiring the partition lock for write op
0810 08:00:38.857899 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0810 08:00:38.857921 (+ 22us) tablet.cc:683] Acquiring locks for 2 operations
0810 08:00:38.872874 (+ 14953us) tablet.cc:699] Row locks acquired
0810 08:00:38.872899 (+ 25us) write_op.cc:260] PREPARE: finished
0810 08:00:38.873061 (+ 162us) write_op.cc:270] Start()
0810 08:00:38.873182 (+ 121us) write_op.cc:276] Timestamp: P: 1786348838872999 usec, L: 0
0810 08:00:38.873205 (+ 23us) op_driver.cc:331] REPLICATION: starting
0810 08:00:38.884389 (+ 11184us) log.cc:842] Serialized 533 byte log entry
0810 08:00:39.156527 (+272138us) op_driver.cc:444] REPLICATION: finished
0810 08:00:39.158353 (+ 1826us) write_op.cc:301] APPLY: starting
0810 08:00:39.158405 (+ 52us) tablet.cc:1365] starting BulkCheckPresence
0810 08:00:39.158526 (+ 121us) tablet.cc:1368] finished BulkCheckPresence
0810 08:00:39.158630 (+ 104us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 08:00:39.158829 (+ 199us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 08:00:39.158958 (+ 129us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0810 08:00:39.158998 (+ 40us) write_op.cc:312] APPLY: finished
0810 08:00:39.159164 (+ 166us) log.cc:842] Serialized 29 byte log entry
0810 08:00:39.159751 (+ 587us) write_op.cc:489] Releasing partition, row and schema locks
0810 08:00:39.159816 (+ 65us) write_op.cc:454] Released schema lock
0810 08:00:39.159847 (+ 31us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":284427520,"child_traces":[["op",{"apply.queue_time_us":1636,"num_ops":2,"prepare.queue_time_us":564313,"prepare.run_cpu_time_us":2564,"prepare.run_wall_time_us":79815,"replication_time_us":283238}]]}
W20260810 08:00:39.166198 14681 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 8980090fc4ff4de68bbb690f30c3f46e (table default.Table_ec141df5dd0d4724a63dd28e7682d9af [id=7835a5751ffe4cac8863cb9aec1e3a87], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8980090fc4ff4de68bbb690f30c3f46e consensus state has no leader
I20260810 08:00:39.175887 14686 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42300 (request call id 259) took 1024 ms. Trace:
I20260810 08:00:39.176225 14686 rpcz_store.cc:276] 0810 08:00:38.151855 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:38.265752 (+113897us) service_pool.cc:227] Handling call
0810 08:00:38.268478 (+ 2726us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 08:00:38.274769 (+ 6291us) catalog_manager.cc:2121] Created new table and tablet info
0810 08:00:39.175254 (+900485us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 08:00:39.175563 (+ 309us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 08:00:39.175826 (+ 263us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 08:00:38.885908 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 08:00:38.886240 (+ 332us) write_op.cc:432] Acquiring schema lock in shared mode
0810 08:00:38.886265 (+ 25us) write_op.cc:435] Acquired schema lock
0810 08:00:38.886281 (+ 16us) tablet.cc:660] Decoding operations
0810 08:00:38.886718 (+ 437us) write_op.cc:620] Acquiring the partition lock for write op
0810 08:00:38.886795 (+ 77us) write_op.cc:641] Partition lock acquired for write op
0810 08:00:38.886829 (+ 34us) tablet.cc:683] Acquiring locks for 2 operations
0810 08:00:38.887076 (+ 247us) tablet.cc:699] Row locks acquired
0810 08:00:38.887096 (+ 20us) write_op.cc:260] PREPARE: finished
0810 08:00:38.887237 (+ 141us) write_op.cc:270] Start()
0810 08:00:38.887357 (+ 120us) write_op.cc:276] Timestamp: P: 1786348838887177 usec, L: 0
0810 08:00:38.887381 (+ 24us) op_driver.cc:331] REPLICATION: starting
0810 08:00:38.888036 (+ 655us) log.cc:842] Serialized 433 byte log entry
0810 08:00:39.156746 (+268710us) op_driver.cc:444] REPLICATION: finished
0810 08:00:39.172381 (+ 15635us) write_op.cc:301] APPLY: starting
0810 08:00:39.172431 (+ 50us) tablet.cc:1365] starting BulkCheckPresence
0810 08:00:39.172542 (+ 111us) tablet.cc:1368] finished BulkCheckPresence
0810 08:00:39.172686 (+ 144us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 08:00:39.173103 (+ 417us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 08:00:39.173739 (+ 636us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 08:00:39.173789 (+ 50us) write_op.cc:312] APPLY: finished
0810 08:00:39.174095 (+ 306us) log.cc:842] Serialized 29 byte log entry
0810 08:00:39.174853 (+ 758us) write_op.cc:489] Releasing partition, row and schema locks
0810 08:00:39.174952 (+ 99us) write_op.cc:454] Released schema lock
0810 08:00:39.175012 (+ 60us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":15507,"num_ops":2,"prepare.queue_time_us":610077,"prepare.run_cpu_time_us":2328,"prepare.run_wall_time_us":3253,"replication_time_us":269295}]]}
I20260810 08:00:39.179540 14686 master_service.cc:946] ReplaceTablet: received request to replace tablet 6173d97ad84c43b3848c1b673d7a35b7 from {username='slave'} at 127.0.0.1:42300
I20260810 08:00:39.189998 14694 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: Sending DeleteTablet for 3 replicas of tablet fcd6d992683c4a5c886d7c5052da76b3
I20260810 08:00:39.191346 14694 catalog_manager.cc:6565] ReplaceTablet: tablet fcd6d992683c4a5c886d7c5052da76b3 deleted and replaced by tablet 7172aab8012a49759dfa80dad3e4cc39
I20260810 08:00:39.191679 14694 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:42300 (request call id 256) took 1284 ms. Trace:
W20260810 08:00:39.193787 14918 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.57.195:37729, user_credentials={real_user=slave}} blocked reactor thread for 115256us
I20260810 08:00:39.194734 15234 leader_election.cc:302] T 8364045e3fa2441e8dc1a1d87f73361d P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224; no voters: 9d7219686f304f62a54acc7497cbbd07
I20260810 08:00:39.200824 15116 tablet_service.cc:1514] Processing DeleteTablet for tablet fcd6d992683c4a5c886d7c5052da76b3 with delete_type TABLET_DATA_DELETED (replaced by tablet 7172aab8012a49759dfa80dad3e4cc39) from {username='slave'} at 127.0.0.1:43002
I20260810 08:00:39.205363 14963 tablet_service.cc:1514] Processing DeleteTablet for tablet fcd6d992683c4a5c886d7c5052da76b3 with delete_type TABLET_DATA_DELETED (replaced by tablet 7172aab8012a49759dfa80dad3e4cc39) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:39.206755 16045 raft_consensus.cc:2781] T 8364045e3fa2441e8dc1a1d87f73361d P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:39.207474 16045 raft_consensus.cc:482] T 8364045e3fa2441e8dc1a1d87f73361d P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:39.208315 16045 raft_consensus.cc:3037] T 8364045e3fa2441e8dc1a1d87f73361d P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:39.213742 15490 tablet_bootstrap.cc:492] T fd97cbfb16bf4ee2b4eed1e8bbbc7224 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:39.214617 15490 ts_tablet_manager.cc:1397] T fd97cbfb16bf4ee2b4eed1e8bbbc7224 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.080s user 0.001s sys 0.009s
I20260810 08:00:39.191965 14694 rpcz_store.cc:276] 0810 08:00:37.906946 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:37.907051 (+ 105us) service_pool.cc:227] Handling call
0810 08:00:39.191626 (+1284575us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 08:00:38.290902 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 08:00:38.291240 (+ 338us) write_op.cc:432] Acquiring schema lock in shared mode
0810 08:00:38.291260 (+ 20us) write_op.cc:435] Acquired schema lock
0810 08:00:38.291271 (+ 11us) tablet.cc:660] Decoding operations
0810 08:00:38.291555 (+ 284us) write_op.cc:620] Acquiring the partition lock for write op
0810 08:00:38.291588 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0810 08:00:38.291608 (+ 20us) tablet.cc:683] Acquiring locks for 2 operations
0810 08:00:38.291798 (+ 190us) tablet.cc:699] Row locks acquired
0810 08:00:38.291813 (+ 15us) write_op.cc:260] PREPARE: finished
0810 08:00:38.291922 (+ 109us) write_op.cc:270] Start()
0810 08:00:38.292015 (+ 93us) write_op.cc:276] Timestamp: P: 1786348838291885 usec, L: 0
0810 08:00:38.292032 (+ 17us) op_driver.cc:331] REPLICATION: starting
0810 08:00:38.292533 (+ 501us) log.cc:842] Serialized 628 byte log entry
0810 08:00:39.057419 (+764886us) op_driver.cc:444] REPLICATION: finished
0810 08:00:39.058864 (+ 1445us) write_op.cc:301] APPLY: starting
0810 08:00:39.058926 (+ 62us) tablet.cc:1365] starting BulkCheckPresence
0810 08:00:39.059037 (+ 111us) tablet.cc:1368] finished BulkCheckPresence
0810 08:00:39.059141 (+ 104us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 08:00:39.059512 (+ 371us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 08:00:39.059716 (+ 204us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 08:00:39.059759 (+ 43us) write_op.cc:312] APPLY: finished
0810 08:00:39.059973 (+ 214us) log.cc:842] Serialized 29 byte log entry
0810 08:00:39.189159 (+129186us) write_op.cc:489] Releasing partition, row and schema locks
0810 08:00:39.189286 (+ 127us) write_op.cc:454] Released schema lock
0810 08:00:39.189350 (+ 64us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":11136,"child_traces":[["op",{"apply.queue_time_us":1178,"num_ops":2,"prepare.queue_time_us":379925,"prepare.run_cpu_time_us":1810,"prepare.run_wall_time_us":1810,"replication_time_us":765310,"thread_start_us":15763,"threads_started":1}]]}
I20260810 08:00:39.220459 14694 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42300:
name: "default.Table_20b0364a393c4e55940c6bc104f48c52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 08:00:39.222400 14694 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20b0364a393c4e55940c6bc104f48c52 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 08:00:39.226861 15490 raft_consensus.cc:348] T fd97cbfb16bf4ee2b4eed1e8bbbc7224 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:39.227547 15490 raft_consensus.cc:374] T fd97cbfb16bf4ee2b4eed1e8bbbc7224 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:39.227808 15490 raft_consensus.cc:729] T fd97cbfb16bf4ee2b4eed1e8bbbc7224 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:39.121480 16054 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:39.228523 15490 consensus_queue.cc:260] T fd97cbfb16bf4ee2b4eed1e8bbbc7224 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:39.233343 14622 raft_consensus.cc:1226] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 3 FOLLOWER]: Rejecting Update request from peer 38a7164890624423a7427871aa2b10e0 for earlier term 2. Current term is 3. Ops: [2.287-2.287]
I20260810 08:00:39.235371 15490 ts_tablet_manager.cc:1428] T fd97cbfb16bf4ee2b4eed1e8bbbc7224 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.020s user 0.005s sys 0.001s
I20260810 08:00:39.239745 15273 tablet_service.cc:1514] Processing DeleteTablet for tablet fcd6d992683c4a5c886d7c5052da76b3 with delete_type TABLET_DATA_DELETED (replaced by tablet 7172aab8012a49759dfa80dad3e4cc39) from {username='slave'} at 127.0.0.1:40692
I20260810 08:00:39.241753 15980 raft_consensus.cc:482] T fd97cbfb16bf4ee2b4eed1e8bbbc7224 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:39.242235 15980 raft_consensus.cc:504] T fd97cbfb16bf4ee2b4eed1e8bbbc7224 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:39.246544 15980 leader_election.cc:288] T fd97cbfb16bf4ee2b4eed1e8bbbc7224 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:39.247648 15437 tablet_bootstrap.cc:492] T 389b30cba2b149e9ab0de8c191808db2 P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:39.248509 15437 ts_tablet_manager.cc:1397] T 389b30cba2b149e9ab0de8c191808db2 P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.385s user 0.016s sys 0.010s
I20260810 08:00:39.248461 15302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd97cbfb16bf4ee2b4eed1e8bbbc7224" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:39.250586 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd97cbfb16bf4ee2b4eed1e8bbbc7224" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
W20260810 08:00:39.250515 15076 leader_election.cc:341] T fd97cbfb16bf4ee2b4eed1e8bbbc7224 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
W20260810 08:00:39.252307 15079 leader_election.cc:341] T fd97cbfb16bf4ee2b4eed1e8bbbc7224 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:39.252671 15079 leader_election.cc:302] T fd97cbfb16bf4ee2b4eed1e8bbbc7224 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:39.253790 15980 raft_consensus.cc:2726] T fd97cbfb16bf4ee2b4eed1e8bbbc7224 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:39.255195 14692 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42300 (request call id 255) took 1378 ms. Trace:
I20260810 08:00:39.255539 14692 rpcz_store.cc:276] 0810 08:00:37.876634 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:37.876773 (+ 139us) service_pool.cc:227] Handling call
0810 08:00:37.894088 (+ 17315us) spinlock_profiling.cc:243] Waited 16.3 ms on lock 0x7b5400021554. stack: 00007fe2372c4033 00007fe2372c3f30 00007fe236e45534 00000000004db2b4 00000000004db249 00007fe23e1d0af0 00007fe23e253f2a 00007fe24011fd6c 00007fe24021b156 00007fe23dd58034 00007fe23dd57fc2 00007fe23dd57ef1 00007fe23dd57e6c 00007fe23dd57122 00007fe23bffa94c 00007fe23bff9d86
0810 08:00:37.896616 (+ 2528us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 08:00:37.898635 (+ 2019us) catalog_manager.cc:2121] Created new table and tablet info
0810 08:00:39.254451 (+1355816us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 08:00:39.254797 (+ 346us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 08:00:39.255137 (+ 340us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 08:00:38.288837 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 08:00:38.289190 (+ 353us) write_op.cc:432] Acquiring schema lock in shared mode
0810 08:00:38.289212 (+ 22us) write_op.cc:435] Acquired schema lock
0810 08:00:38.289227 (+ 15us) tablet.cc:660] Decoding operations
0810 08:00:38.289500 (+ 273us) write_op.cc:620] Acquiring the partition lock for write op
0810 08:00:38.289538 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0810 08:00:38.289559 (+ 21us) tablet.cc:683] Acquiring locks for 2 operations
0810 08:00:38.289774 (+ 215us) tablet.cc:699] Row locks acquired
0810 08:00:38.289788 (+ 14us) write_op.cc:260] PREPARE: finished
0810 08:00:38.289904 (+ 116us) write_op.cc:270] Start()
0810 08:00:38.289986 (+ 82us) write_op.cc:276] Timestamp: P: 1786348838289866 usec, L: 0
0810 08:00:38.290002 (+ 16us) op_driver.cc:331] REPLICATION: starting
0810 08:00:38.290594 (+ 592us) log.cc:842] Serialized 433 byte log entry
0810 08:00:39.031439 (+740845us) op_driver.cc:444] REPLICATION: finished
0810 08:00:39.050355 (+ 18916us) write_op.cc:301] APPLY: starting
0810 08:00:39.050424 (+ 69us) tablet.cc:1365] starting BulkCheckPresence
0810 08:00:39.050541 (+ 117us) tablet.cc:1368] finished BulkCheckPresence
0810 08:00:39.050644 (+ 103us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 08:00:39.051103 (+ 459us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 08:00:39.051283 (+ 180us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 08:00:39.051331 (+ 48us) write_op.cc:312] APPLY: finished
0810 08:00:39.051539 (+ 208us) log.cc:842] Serialized 29 byte log entry
0810 08:00:39.239053 (+187514us) write_op.cc:489] Releasing partition, row and schema locks
0810 08:00:39.239177 (+ 124us) write_op.cc:454] Released schema lock
0810 08:00:39.239239 (+ 62us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":37559680,"child_traces":[["op",{"apply.queue_time_us":18685,"num_ops":2,"prepare.queue_time_us":388665,"prepare.run_cpu_time_us":1935,"prepare.run_wall_time_us":1965,"replication_time_us":741346,"thread_start_us":25616,"threads_started":1}]]}
I20260810 08:00:39.263556 15490 tablet_bootstrap.cc:492] T 20a316e404414b288a315a5386208111 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:39.279414 16056 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 }, Status: INVALID_TERM, Last received: 2.286, Next index: 287, Last known committed idx: 286, Time since last communication: 0.001s
I20260810 08:00:39.282079 16055 raft_consensus.cc:3032] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 2 LEADER]: Stepping down as leader of term 2
I20260810 08:00:39.282725 16055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 38a7164890624423a7427871aa2b10e0, State: Running, Role: LEADER
I20260810 08:00:39.288853 15439 tablet_bootstrap.cc:654] T 579bac622ddf40bf80d01d66ae2fa100 P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:39.284258 16055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 286, Committed index: 286, Last appended: 2.289, Last appended by leader: 289, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:39.289846 16055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 2 FOLLOWER]: Advancing to term 3
W20260810 08:00:39.300200 14681 catalog_manager.cc:4821] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): delete failed for tablet fcd6d992683c4a5c886d7c5052da76b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fcd6d992683c4a5c886d7c5052da76b3 already in progress: creating tablet
I20260810 08:00:39.306413 14705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99f2d868d76c440f8443b33e1cd19f58" candidate_term: 3 candidate_status { last_received { term: 2 index: 286 } } ignore_live_leader: false dest_uuid: "38a7164890624423a7427871aa2b10e0"
I20260810 08:00:39.312996 16054 leader_election.cc:288] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [CANDIDATE]: Term 3 election: Requested vote from peers 38a7164890624423a7427871aa2b10e0 (127.14.57.253:40439), 1e8cadb26fe0428a964f66ef8172f3be (127.14.57.252:33819)
I20260810 08:00:39.313993 15797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99f2d868d76c440f8443b33e1cd19f58" candidate_term: 3 candidate_status { last_received { term: 2 index: 286 } } ignore_live_leader: false dest_uuid: "1e8cadb26fe0428a964f66ef8172f3be"
I20260810 08:00:39.314744 15797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [term 2 FOLLOWER]: Advancing to term 3
W20260810 08:00:39.317340 14689 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0: An error occurred while updating sys-tables: Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.062s ago.
I20260810 08:00:39.338363 15490 tablet_bootstrap.cc:654] T 20a316e404414b288a315a5386208111 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:39.341491 16023 raft_consensus.cc:482] T 924c0f73638f48a88c08d45d459132e1 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:39.342219 16023 raft_consensus.cc:504] T 924c0f73638f48a88c08d45d459132e1 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:39.344158 15437 raft_consensus.cc:348] T 389b30cba2b149e9ab0de8c191808db2 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:39.344866 15437 raft_consensus.cc:374] T 389b30cba2b149e9ab0de8c191808db2 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:39.345077 15437 raft_consensus.cc:729] T 389b30cba2b149e9ab0de8c191808db2 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:39.346858 15797 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99f2d868d76c440f8443b33e1cd19f58 in term 3.
I20260810 08:00:39.349050 14598 leader_election.cc:302] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e8cadb26fe0428a964f66ef8172f3be, 99f2d868d76c440f8443b33e1cd19f58; no voters:
I20260810 08:00:39.360582 16023 leader_election.cc:288] T 924c0f73638f48a88c08d45d459132e1 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:39.364326 16054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 3 FOLLOWER]: Leader election won for term 3
W20260810 08:00:39.375535 14918 leader_election.cc:341] T 924c0f73638f48a88c08d45d459132e1 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Not found: Tablet not found: 924c0f73638f48a88c08d45d459132e1
I20260810 08:00:39.380833 14705 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 99f2d868d76c440f8443b33e1cd19f58 for term 3 because replica has last-logged OpId of term: 2 index: 289, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 286.
W20260810 08:00:39.386583 14922 leader_election.cc:341] T 924c0f73638f48a88c08d45d459132e1 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Not found: Tablet not found: 924c0f73638f48a88c08d45d459132e1
I20260810 08:00:39.387038 14922 leader_election.cc:302] T 924c0f73638f48a88c08d45d459132e1 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a91b27762182454baada71e91ceab224; no voters: 3fc5e568f3724f158b616c303efb9eae, 9d7219686f304f62a54acc7497cbbd07
I20260810 08:00:39.345577 15437 consensus_queue.cc:260] T 389b30cba2b149e9ab0de8c191808db2 P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
W20260810 08:00:39.394609 15366 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.14.57.253:40439)
I20260810 08:00:39.397840 15490 tablet_bootstrap.cc:492] T 20a316e404414b288a315a5386208111 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:39.397661 16070 raft_consensus.cc:2726] T 924c0f73638f48a88c08d45d459132e1 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 08:00:39.398365 15490 ts_tablet_manager.cc:1397] T 20a316e404414b288a315a5386208111 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.135s user 0.012s sys 0.005s
W20260810 08:00:39.399665 15366 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.14.57.253:40439)
I20260810 08:00:39.400288 15437 ts_tablet_manager.cc:1428] T 389b30cba2b149e9ab0de8c191808db2 P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.151s user 0.003s sys 0.004s
W20260810 08:00:39.410874 15366 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.14.57.253:40439)
I20260810 08:00:39.326519 15890 ts_tablet_manager.cc:1918] T 2d295305b5a843eaa84c8ad666355b84 P a91b27762182454baada71e91ceab224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 08:00:39.416143 15890 log.cc:1196] T 2d295305b5a843eaa84c8ad666355b84 P a91b27762182454baada71e91ceab224: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal/wals/2d295305b5a843eaa84c8ad666355b84
I20260810 08:00:39.419767 16045 raft_consensus.cc:504] T 8364045e3fa2441e8dc1a1d87f73361d P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:39.420610 16054 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 3 LEADER]: Becoming Leader. State: Replica: 99f2d868d76c440f8443b33e1cd19f58, State: Running, Role: LEADER
I20260810 08:00:39.422649 15890 ts_tablet_manager.cc:1939] T 2d295305b5a843eaa84c8ad666355b84 P a91b27762182454baada71e91ceab224: Deleting consensus metadata
I20260810 08:00:39.422465 16054 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 286, Committed index: 286, Last appended: 2.286, Last appended by leader: 286, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:39.426199 15490 raft_consensus.cc:348] T 20a316e404414b288a315a5386208111 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:39.436966 15490 raft_consensus.cc:374] T 20a316e404414b288a315a5386208111 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:39.437295 15490 raft_consensus.cc:729] T 20a316e404414b288a315a5386208111 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:39.438040 15490 consensus_queue.cc:260] T 20a316e404414b288a315a5386208111 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:39.428587 15437 tablet_bootstrap.cc:492] T 3ac554bd37b045a2a8cf2743f4645f28 P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:39.481873 15490 ts_tablet_manager.cc:1428] T 20a316e404414b288a315a5386208111 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.083s user 0.009s sys 0.000s
I20260810 08:00:39.473562 15437 tablet_bootstrap.cc:654] T 3ac554bd37b045a2a8cf2743f4645f28 P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:39.493858 16082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99f2d868d76c440f8443b33e1cd19f58. Latest consensus state: current_term: 3 leader_uuid: "99f2d868d76c440f8443b33e1cd19f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } } }
I20260810 08:00:39.499687 15490 tablet_bootstrap.cc:492] T abc607ab01f74b2c9c693d6dea73090c P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:39.508652 15439 tablet_bootstrap.cc:492] T 579bac622ddf40bf80d01d66ae2fa100 P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:39.518461 15439 ts_tablet_manager.cc:1397] T 579bac622ddf40bf80d01d66ae2fa100 P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.492s user 0.007s sys 0.006s
I20260810 08:00:39.520704 15437 tablet_bootstrap.cc:492] T 3ac554bd37b045a2a8cf2743f4645f28 P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:39.521327 15437 ts_tablet_manager.cc:1397] T 3ac554bd37b045a2a8cf2743f4645f28 P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.093s user 0.012s sys 0.007s
I20260810 08:00:39.526374 16045 leader_election.cc:288] T 8364045e3fa2441e8dc1a1d87f73361d P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Requested vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:39.527583 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8364045e3fa2441e8dc1a1d87f73361d" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224"
I20260810 08:00:39.528468 14987 raft_consensus.cc:3037] T 8364045e3fa2441e8dc1a1d87f73361d P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:39.532298 15437 raft_consensus.cc:348] T 3ac554bd37b045a2a8cf2743f4645f28 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:39.533111 15437 raft_consensus.cc:374] T 3ac554bd37b045a2a8cf2743f4645f28 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:39.533403 15437 raft_consensus.cc:729] T 3ac554bd37b045a2a8cf2743f4645f28 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:39.539407 15437 consensus_queue.cc:260] T 3ac554bd37b045a2a8cf2743f4645f28 P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:39.543421 15490 tablet_bootstrap.cc:654] T abc607ab01f74b2c9c693d6dea73090c P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:39.549829 15437 ts_tablet_manager.cc:1428] T 3ac554bd37b045a2a8cf2743f4645f28 P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.028s user 0.008s sys 0.000s
I20260810 08:00:39.555035 16070 raft_consensus.cc:482] T e0652a7684b04a89840049ce0d781199 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:39.555796 16070 raft_consensus.cc:504] T e0652a7684b04a89840049ce0d781199 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:39.559984 16070 leader_election.cc:288] T e0652a7684b04a89840049ce0d781199 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:39.562191 15145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0652a7684b04a89840049ce0d781199" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:39.563251 15145 raft_consensus.cc:2454] T e0652a7684b04a89840049ce0d781199 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 0.
I20260810 08:00:39.567265 15980 raft_consensus.cc:482] T ae51ddeb22fe4cf7a8aa2d8b2b70a535 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:39.567880 15980 raft_consensus.cc:504] T ae51ddeb22fe4cf7a8aa2d8b2b70a535 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:39.571933 15980 leader_election.cc:288] T ae51ddeb22fe4cf7a8aa2d8b2b70a535 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:39.574024 14988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae51ddeb22fe4cf7a8aa2d8b2b70a535" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
W20260810 08:00:39.576186 15079 leader_election.cc:341] T ae51ddeb22fe4cf7a8aa2d8b2b70a535 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:39.566723 15437 tablet_bootstrap.cc:492] T e13b91d61063446f9d1454da4f8bb706 P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:39.561342 15890 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34630 (request call id 154) took 5026 ms. Trace:
I20260810 08:00:39.565042 14922 leader_election.cc:302] T e0652a7684b04a89840049ce0d781199 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224; no voters:
I20260810 08:00:39.595273 16082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [sys.catalog]: This master's current role is: LEADER
W20260810 08:00:39.534222 15234 leader_election.cc:341] T 8364045e3fa2441e8dc1a1d87f73361d P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Not found: Tablet not found: 8364045e3fa2441e8dc1a1d87f73361d
I20260810 08:00:39.538278 15439 raft_consensus.cc:348] T 579bac622ddf40bf80d01d66ae2fa100 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:39.596628 15439 raft_consensus.cc:374] T 579bac622ddf40bf80d01d66ae2fa100 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260810 08:00:39.596673 15234 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.194:36265, user_credentials={real_user=slave}} blocked reactor thread for 62483.3us
I20260810 08:00:39.596983 15439 raft_consensus.cc:729] T 579bac622ddf40bf80d01d66ae2fa100 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:39.598332 15298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae51ddeb22fe4cf7a8aa2d8b2b70a535" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:39.599255 15439 consensus_queue.cc:260] T 579bac622ddf40bf80d01d66ae2fa100 P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
W20260810 08:00:39.599922 15076 leader_election.cc:341] T ae51ddeb22fe4cf7a8aa2d8b2b70a535 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:39.600574 15076 leader_election.cc:302] T ae51ddeb22fe4cf7a8aa2d8b2b70a535 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:39.601822 15439 ts_tablet_manager.cc:1428] T 579bac622ddf40bf80d01d66ae2fa100 P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.077s user 0.005s sys 0.004s
I20260810 08:00:39.602497 15980 raft_consensus.cc:2726] T ae51ddeb22fe4cf7a8aa2d8b2b70a535 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:39.602854 15439 tablet_bootstrap.cc:492] T e0652a7684b04a89840049ce0d781199 P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:39.608126 15439 tablet_bootstrap.cc:654] T e0652a7684b04a89840049ce0d781199 P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:39.614614 15958 raft_consensus.cc:2781] T e0652a7684b04a89840049ce0d781199 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:39.615005 15958 raft_consensus.cc:482] T e0652a7684b04a89840049ce0d781199 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:39.615304 15958 raft_consensus.cc:3037] T e0652a7684b04a89840049ce0d781199 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:39.618088 15890 rpcz_store.cc:276] 0810 08:00:34.535095 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:34.537734 (+ 2639us) service_pool.cc:227] Handling call
0810 08:00:38.671441 (+4133707us) ts_tablet_manager.cc:1145] Deleting tablet 2d295305b5a843eaa84c8ad666355b84
0810 08:00:38.671567 (+ 126us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:38.760428 (+ 88861us) spinlock_profiling.cc:243] Waited 40.7 ms on lock 0x7b280003d388. stack: 00007feaedce8033 00007feaedce7f30 00007feaed869534 00000000004db2b4 00000000004db249 00007feaf6b608b1 00007feaf6b741ea 00007feaf4fe049f 00007feaf508d3fb 00007feaf5c9acfa 00007feaf5ccae10 00007feaf5cb1e0d 00007feaf5cb1db9 00007feaf5cb1d49 00007feaf5cb1d11 00007feaf5cb0f3d
0810 08:00:38.902020 (+141592us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:39.297184 (+395164us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:39.561264 (+264080us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":93683584,"tablet-delete.queue_time_us":4133105}
I20260810 08:00:39.621099 15890 tablet_replica.cc:330] T 924c0f73638f48a88c08d45d459132e1 P a91b27762182454baada71e91ceab224: stopping tablet replica
I20260810 08:00:39.631587 15890 raft_consensus.cc:2229] T 924c0f73638f48a88c08d45d459132e1 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 08:00:39.632532 15890 raft_consensus.cc:2258] T 924c0f73638f48a88c08d45d459132e1 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:39.636309 15890 ts_tablet_manager.cc:1905] T 924c0f73638f48a88c08d45d459132e1 P a91b27762182454baada71e91ceab224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:39.647498 15490 tablet_bootstrap.cc:492] T abc607ab01f74b2c9c693d6dea73090c P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:39.648022 15490 ts_tablet_manager.cc:1397] T abc607ab01f74b2c9c693d6dea73090c P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.149s user 0.016s sys 0.041s
I20260810 08:00:39.650205 15490 raft_consensus.cc:348] T abc607ab01f74b2c9c693d6dea73090c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:39.650872 15490 raft_consensus.cc:374] T abc607ab01f74b2c9c693d6dea73090c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:39.651126 15490 raft_consensus.cc:729] T abc607ab01f74b2c9c693d6dea73090c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:39.651852 15490 consensus_queue.cc:260] T abc607ab01f74b2c9c693d6dea73090c P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:39.629115 14987 raft_consensus.cc:2454] T 8364045e3fa2441e8dc1a1d87f73361d P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 1.
I20260810 08:00:39.650632 15437 tablet_bootstrap.cc:654] T e13b91d61063446f9d1454da4f8bb706 P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:39.576314 16002 ts_tablet_manager.cc:1918] T 1f427f4093e5417a9ca79b1805b0675e P 3fc5e568f3724f158b616c303efb9eae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 08:00:39.565207 15302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0652a7684b04a89840049ce0d781199" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:39.656172 16089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 08:00:39.656965 16002 log.cc:1196] T 1f427f4093e5417a9ca79b1805b0675e P 3fc5e568f3724f158b616c303efb9eae: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-2/wal/wals/1f427f4093e5417a9ca79b1805b0675e
I20260810 08:00:39.657951 16002 ts_tablet_manager.cc:1939] T 1f427f4093e5417a9ca79b1805b0675e P 3fc5e568f3724f158b616c303efb9eae: Deleting consensus metadata
I20260810 08:00:39.660416 16002 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 192) took 2078 ms. Trace:
W20260810 08:00:39.656931 14918 leader_election.cc:341] T e0652a7684b04a89840049ce0d781199 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:39.667479 16002 rpcz_store.cc:276] 0810 08:00:37.582510 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:37.585748 (+ 3238us) service_pool.cc:227] Handling call
0810 08:00:38.678033 (+1092285us) ts_tablet_manager.cc:1145] Deleting tablet 1f427f4093e5417a9ca79b1805b0675e
0810 08:00:38.678119 (+ 86us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:39.237827 (+559708us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:39.575893 (+338066us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:39.660352 (+ 84459us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1004057}
I20260810 08:00:39.661856 15234 leader_election.cc:302] T 8364045e3fa2441e8dc1a1d87f73361d P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224; no voters: 9d7219686f304f62a54acc7497cbbd07
I20260810 08:00:39.667212 15439 tablet_bootstrap.cc:492] T e0652a7684b04a89840049ce0d781199 P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:39.671541 15439 ts_tablet_manager.cc:1397] T e0652a7684b04a89840049ce0d781199 P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.069s user 0.004s sys 0.005s
I20260810 08:00:39.672183 16045 raft_consensus.cc:2781] T 8364045e3fa2441e8dc1a1d87f73361d P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Leader election won for term 1
I20260810 08:00:39.672616 16045 raft_consensus.cc:686] T 8364045e3fa2441e8dc1a1d87f73361d P 3fc5e568f3724f158b616c303efb9eae [term 1 LEADER]: Becoming Leader. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Running, Role: LEADER
I20260810 08:00:39.673336 16045 consensus_queue.cc:237] T 8364045e3fa2441e8dc1a1d87f73361d P 3fc5e568f3724f158b616c303efb9eae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:39.673805 15439 raft_consensus.cc:348] T e0652a7684b04a89840049ce0d781199 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:39.674517 15439 raft_consensus.cc:374] T e0652a7684b04a89840049ce0d781199 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:39.675164 15439 raft_consensus.cc:729] T e0652a7684b04a89840049ce0d781199 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:39.669902 16002 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 193) took 1914 ms. Trace:
I20260810 08:00:39.676286 16002 rpcz_store.cc:276] 0810 08:00:37.755497 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:37.755624 (+ 127us) service_pool.cc:227] Handling call
0810 08:00:39.669491 (+1913867us) ts_tablet_manager.cc:1145] Deleting tablet 75681d70a3e742768df36a1eaacc3658
0810 08:00:39.669542 (+ 51us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:39.669850 (+ 308us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1912623}
I20260810 08:00:39.641314 15958 raft_consensus.cc:504] T e0652a7684b04a89840049ce0d781199 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:39.684609 15145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0652a7684b04a89840049ce0d781199" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07"
I20260810 08:00:39.684912 15490 ts_tablet_manager.cc:1428] T abc607ab01f74b2c9c693d6dea73090c P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.037s user 0.005s sys 0.001s
I20260810 08:00:39.685214 15145 raft_consensus.cc:3037] T e0652a7684b04a89840049ce0d781199 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:39.701797 15958 leader_election.cc:288] T e0652a7684b04a89840049ce0d781199 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Requested vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:39.675719 15439 consensus_queue.cc:260] T e0652a7684b04a89840049ce0d781199 P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:39.713603 15439 ts_tablet_manager.cc:1428] T e0652a7684b04a89840049ce0d781199 P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.042s user 0.005s sys 0.000s
I20260810 08:00:39.714282 15145 raft_consensus.cc:2454] T e0652a7684b04a89840049ce0d781199 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 1.
I20260810 08:00:39.714388 14681 catalog_manager.cc:4841] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): tablet 2d295305b5a843eaa84c8ad666355b84 (table default.Table_d5c08115c8b24ad48a9686e48a45ab82 [id=bb6d5450013a45b1bb6a85bc9e4e6d9c]) successfully deleted
I20260810 08:00:39.714241 14678 catalog_manager.cc:4841] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): tablet 1f427f4093e5417a9ca79b1805b0675e (table default.Table_e0d4352887e04b38939d453e07a37ef9 [id=11ca1b81b7704d33820eef06f5a25310]) successfully deleted
W20260810 08:00:39.716495 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet 75681d70a3e742768df36a1eaacc3658 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75681d70a3e742768df36a1eaacc3658 already in progress: creating tablet
I20260810 08:00:39.728313 14922 leader_election.cc:302] T e0652a7684b04a89840049ce0d781199 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224; no voters:
I20260810 08:00:39.729278 15958 raft_consensus.cc:2781] T e0652a7684b04a89840049ce0d781199 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 08:00:39.729796 15958 raft_consensus.cc:686] T e0652a7684b04a89840049ce0d781199 P a91b27762182454baada71e91ceab224 [term 1 LEADER]: Becoming Leader. State: Replica: a91b27762182454baada71e91ceab224, State: Running, Role: LEADER
I20260810 08:00:39.730643 15958 consensus_queue.cc:237] T e0652a7684b04a89840049ce0d781199 P a91b27762182454baada71e91ceab224 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:39.734535 15490 tablet_bootstrap.cc:492] T b18980bffe6045b5b69bcecb47b4c711 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:39.736415 15439 tablet_bootstrap.cc:492] T adada21bc1bf4f19a6347d0935bd74bd P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:39.685359 15302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0652a7684b04a89840049ce0d781199" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae"
I20260810 08:00:39.737902 15302 raft_consensus.cc:3037] T e0652a7684b04a89840049ce0d781199 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:39.742017 15490 tablet_bootstrap.cc:654] T b18980bffe6045b5b69bcecb47b4c711 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:39.765859 15437 tablet_bootstrap.cc:492] T e13b91d61063446f9d1454da4f8bb706 P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:39.766331 15437 ts_tablet_manager.cc:1397] T e13b91d61063446f9d1454da4f8bb706 P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.200s user 0.012s sys 0.006s
I20260810 08:00:39.767067 15302 raft_consensus.cc:2454] T e0652a7684b04a89840049ce0d781199 P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 1.
I20260810 08:00:39.777779 15437 raft_consensus.cc:348] T e13b91d61063446f9d1454da4f8bb706 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:39.778481 15437 raft_consensus.cc:374] T e13b91d61063446f9d1454da4f8bb706 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:39.778765 15437 raft_consensus.cc:729] T e13b91d61063446f9d1454da4f8bb706 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:39.779412 15437 consensus_queue.cc:260] T e13b91d61063446f9d1454da4f8bb706 P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:39.780366 16055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:39.780961 16055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:39.786103 15797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "38a7164890624423a7427871aa2b10e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 289 } } ignore_live_leader: false dest_uuid: "1e8cadb26fe0428a964f66ef8172f3be" is_pre_election: true
I20260810 08:00:39.786765 15797 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38a7164890624423a7427871aa2b10e0 in term 3.
I20260810 08:00:39.788581 14681 leader_election.cc:302] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e8cadb26fe0428a964f66ef8172f3be, 38a7164890624423a7427871aa2b10e0; no voters:
I20260810 08:00:39.789438 16055 leader_election.cc:288] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 99f2d868d76c440f8443b33e1cd19f58 (127.14.57.254:43757), 1e8cadb26fe0428a964f66ef8172f3be (127.14.57.252:33819)
I20260810 08:00:39.790793 15980 raft_consensus.cc:482] T a92d01d83dbf4b6ba47464dcefa8b2ae P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:39.791273 15980 raft_consensus.cc:504] T a92d01d83dbf4b6ba47464dcefa8b2ae P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:39.793237 15980 leader_election.cc:288] T a92d01d83dbf4b6ba47464dcefa8b2ae P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:39.790086 16055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 08:00:39.793853 16055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:39.794317 16055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 3 FOLLOWER]: Advancing to term 4
I20260810 08:00:39.794519 15437 ts_tablet_manager.cc:1428] T e13b91d61063446f9d1454da4f8bb706 P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.028s user 0.007s sys 0.000s
I20260810 08:00:39.795856 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a92d01d83dbf4b6ba47464dcefa8b2ae" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
W20260810 08:00:39.797936 15079 leader_election.cc:341] T a92d01d83dbf4b6ba47464dcefa8b2ae P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:39.797741 15439 tablet_bootstrap.cc:654] T adada21bc1bf4f19a6347d0935bd74bd P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:39.803383 15490 tablet_bootstrap.cc:492] T b18980bffe6045b5b69bcecb47b4c711 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:39.803788 15490 ts_tablet_manager.cc:1397] T b18980bffe6045b5b69bcecb47b4c711 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.069s user 0.005s sys 0.021s
I20260810 08:00:39.805922 15490 raft_consensus.cc:348] T b18980bffe6045b5b69bcecb47b4c711 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:39.806566 15490 raft_consensus.cc:374] T b18980bffe6045b5b69bcecb47b4c711 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:39.806826 15490 raft_consensus.cc:729] T b18980bffe6045b5b69bcecb47b4c711 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:39.807494 15490 consensus_queue.cc:260] T b18980bffe6045b5b69bcecb47b4c711 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:39.808825 15437 tablet_bootstrap.cc:492] T 8e603d04f7454501b2edfb5a1f4ea7f0 P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:39.815994 14692 master_service.cc:946] ReplaceTablet: received request to replace tablet a4a78b2a960d4d78aa1f48a9c9edf3be from {username='slave'} at 127.0.0.1:42300
I20260810 08:00:39.834666 15490 ts_tablet_manager.cc:1428] T b18980bffe6045b5b69bcecb47b4c711 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.031s user 0.004s sys 0.002s
I20260810 08:00:39.854411 15490 tablet_bootstrap.cc:492] T 3855a2b4d90a47ad918c7a90cc8c7e4a P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:39.872648 15797 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [term 3 FOLLOWER]: Refusing update from remote peer 99f2d868d76c440f8443b33e1cd19f58: Log matching property violated. Preceding OpId in replica: term: 2 index: 181. Preceding OpId from leader: term: 3 index: 287. (index mismatch)
I20260810 08:00:39.876641 16082 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 287, Last known committed idx: 180, Time since last communication: 0.001s
I20260810 08:00:39.881994 15958 raft_consensus.cc:482] T e13b91d61063446f9d1454da4f8bb706 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:39.883041 15490 tablet_bootstrap.cc:654] T 3855a2b4d90a47ad918c7a90cc8c7e4a P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:39.882666 15958 raft_consensus.cc:504] T e13b91d61063446f9d1454da4f8bb706 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:39.887091 15958 leader_election.cc:288] T e13b91d61063446f9d1454da4f8bb706 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:39.887351 15145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13b91d61063446f9d1454da4f8bb706" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:39.887974 15145 raft_consensus.cc:2454] T e13b91d61063446f9d1454da4f8bb706 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 0.
I20260810 08:00:39.889158 14922 leader_election.cc:302] T e13b91d61063446f9d1454da4f8bb706 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224; no voters:
I20260810 08:00:39.890533 15958 raft_consensus.cc:2781] T e13b91d61063446f9d1454da4f8bb706 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:39.890918 15958 raft_consensus.cc:482] T e13b91d61063446f9d1454da4f8bb706 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:39.891309 15958 raft_consensus.cc:3037] T e13b91d61063446f9d1454da4f8bb706 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:39.891247 15970 mvcc.cc:205] Tried to move back new op lower bound from 7316884817690509312 to 7316884774035611648. Current Snapshot: MvccSnapshot[applied={T|T < 7316884817690509312 or (T in {7316884817690509312})}]
I20260810 08:00:39.893402 15980 raft_consensus.cc:482] T 4ca688b4031c41a9884829fbd0be1f33 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:39.893893 15980 raft_consensus.cc:504] T 4ca688b4031c41a9884829fbd0be1f33 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:39.895780 15980 leader_election.cc:288] T 4ca688b4031c41a9884829fbd0be1f33 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:39.896157 15437 tablet_bootstrap.cc:654] T 8e603d04f7454501b2edfb5a1f4ea7f0 P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:39.897682 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ca688b4031c41a9884829fbd0be1f33" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
W20260810 08:00:39.899148 15079 leader_election.cc:341] T 4ca688b4031c41a9884829fbd0be1f33 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:39.901330 16055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
W20260810 08:00:39.905182 15366 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.14.57.253:40439)
I20260810 08:00:39.910233 15890 ts_tablet_manager.cc:1918] T 924c0f73638f48a88c08d45d459132e1 P a91b27762182454baada71e91ceab224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:39.910668 15890 log.cc:1196] T 924c0f73638f48a88c08d45d459132e1 P a91b27762182454baada71e91ceab224: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal/wals/924c0f73638f48a88c08d45d459132e1
I20260810 08:00:39.911350 15890 ts_tablet_manager.cc:1939] T 924c0f73638f48a88c08d45d459132e1 P a91b27762182454baada71e91ceab224: Deleting consensus metadata
I20260810 08:00:39.927407 15490 tablet_bootstrap.cc:492] T 3855a2b4d90a47ad918c7a90cc8c7e4a P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:39.927924 15490 ts_tablet_manager.cc:1397] T 3855a2b4d90a47ad918c7a90cc8c7e4a P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.074s user 0.008s sys 0.001s
I20260810 08:00:39.929936 15980 raft_consensus.cc:482] T 20a316e404414b288a315a5386208111 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:39.930692 15980 raft_consensus.cc:504] T 20a316e404414b288a315a5386208111 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:39.932963 15980 leader_election.cc:288] T 20a316e404414b288a315a5386208111 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:39.933822 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20a316e404414b288a315a5386208111" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:39.934592 15490 raft_consensus.cc:348] T 3855a2b4d90a47ad918c7a90cc8c7e4a P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:39.935249 15490 raft_consensus.cc:374] T 3855a2b4d90a47ad918c7a90cc8c7e4a P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260810 08:00:39.935191 15079 leader_election.cc:341] T 20a316e404414b288a315a5386208111 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:39.935555 15490 raft_consensus.cc:729] T 3855a2b4d90a47ad918c7a90cc8c7e4a P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:39.936249 15490 consensus_queue.cc:260] T 3855a2b4d90a47ad918c7a90cc8c7e4a P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:39.939324 15490 ts_tablet_manager.cc:1428] T 3855a2b4d90a47ad918c7a90cc8c7e4a P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
I20260810 08:00:39.913522 15890 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34630 (request call id 157) took 5259 ms. Trace:
I20260810 08:00:39.940052 15890 rpcz_store.cc:276] 0810 08:00:34.653912 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:34.654026 (+ 114us) service_pool.cc:227] Handling call
0810 08:00:39.620784 (+4966758us) ts_tablet_manager.cc:1145] Deleting tablet 924c0f73638f48a88c08d45d459132e1
0810 08:00:39.620904 (+ 120us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:39.782818 (+161914us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:39.909966 (+127148us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:39.913461 (+ 3495us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5891,"tablet-delete.queue_time_us":4966135}
I20260810 08:00:39.941865 14681 catalog_manager.cc:4841] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): tablet 924c0f73638f48a88c08d45d459132e1 (table default.Table_409addd050ee45b981bddfacfab24c03 [id=76166bd7cb584067b7d2e305603a840b]) successfully deleted
I20260810 08:00:39.945012 15890 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34630 (request call id 159) took 5081 ms. Trace:
I20260810 08:00:39.945456 14705 raft_consensus.cc:1226] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 4 FOLLOWER]: Rejecting Update request from peer 99f2d868d76c440f8443b33e1cd19f58 for earlier term 3. Current term is 4. Ops: []
I20260810 08:00:39.946245 15490 tablet_bootstrap.cc:492] T 8b5a629a7f8041179a006fa412fb6b48 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:39.947355 16054 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 }, Status: INVALID_TERM, Last received: 0.0, Next index: 287, Last known committed idx: 286, Time since last communication: 0.001s
I20260810 08:00:39.961982 15797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "38a7164890624423a7427871aa2b10e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 289 } } ignore_live_leader: false dest_uuid: "1e8cadb26fe0428a964f66ef8172f3be"
I20260810 08:00:39.962616 15797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [term 3 FOLLOWER]: Advancing to term 4
I20260810 08:00:39.970340 15797 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38a7164890624423a7427871aa2b10e0 in term 4.
I20260810 08:00:39.971439 14681 leader_election.cc:302] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e8cadb26fe0428a964f66ef8172f3be, 38a7164890624423a7427871aa2b10e0; no voters:
I20260810 08:00:39.972573 16056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 4 FOLLOWER]: Leader election won for term 4
I20260810 08:00:39.964331 16055 leader_election.cc:288] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [CANDIDATE]: Term 4 election: Requested vote from peers 99f2d868d76c440f8443b33e1cd19f58 (127.14.57.254:43757), 1e8cadb26fe0428a964f66ef8172f3be (127.14.57.252:33819)
I20260810 08:00:39.993535 15980 raft_consensus.cc:482] T 461de46896ee492b9f7cd1202208c19a P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:39.994087 15980 raft_consensus.cc:504] T 461de46896ee492b9f7cd1202208c19a P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:39.996222 15980 leader_election.cc:288] T 461de46896ee492b9f7cd1202208c19a P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:40.001266 15300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ca688b4031c41a9884829fbd0be1f33" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:40.002621 15297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461de46896ee492b9f7cd1202208c19a" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:40.004114 15298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13b91d61063446f9d1454da4f8bb706" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:40.006656 15299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20a316e404414b288a315a5386208111" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:40.006647 15890 rpcz_store.cc:276] 0810 08:00:34.863155 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:34.863271 (+ 116us) service_pool.cc:227] Handling call
0810 08:00:39.944470 (+5081199us) ts_tablet_manager.cc:1145] Deleting tablet 5f2e51d10e7142938e03b2ef687b0edd
0810 08:00:39.944549 (+ 79us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:39.944945 (+ 396us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5080594}
I20260810 08:00:40.008249 15890 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34630 (request call id 158) took 5216 ms. Trace:
W20260810 08:00:40.008538 14681 catalog_manager.cc:4821] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet 5f2e51d10e7142938e03b2ef687b0edd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f2e51d10e7142938e03b2ef687b0edd already in progress: creating tablet
I20260810 08:00:40.008606 15890 rpcz_store.cc:276] 0810 08:00:34.791824 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:34.993808 (+201984us) service_pool.cc:227] Handling call
0810 08:00:40.007784 (+5013976us) ts_tablet_manager.cc:1145] Deleting tablet 0a60001d76604d5085c843961d52213f
0810 08:00:40.007862 (+ 78us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:40.008191 (+ 329us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5009665}
I20260810 08:00:40.011682 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a92d01d83dbf4b6ba47464dcefa8b2ae" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
W20260810 08:00:40.014480 15076 leader_election.cc:341] T 4ca688b4031c41a9884829fbd0be1f33 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:40.015367 15076 leader_election.cc:302] T 4ca688b4031c41a9884829fbd0be1f33 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:39.715556 16002 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40692 (request call id 198) took 1806 ms. Trace:
I20260810 08:00:40.017342 16002 rpcz_store.cc:276] 0810 08:00:37.908764 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:37.909744 (+ 980us) service_pool.cc:227] Handling call
0810 08:00:39.715109 (+1805365us) ts_tablet_manager.cc:1145] Deleting tablet a0887c4efd0f4c7c95e9d12734184a40
0810 08:00:39.715155 (+ 46us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:39.715494 (+ 339us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1704625}
I20260810 08:00:40.024070 15980 raft_consensus.cc:2726] T 4ca688b4031c41a9884829fbd0be1f33 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 08:00:40.016826 15076 leader_election.cc:341] T 461de46896ee492b9f7cd1202208c19a P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
W20260810 08:00:40.026003 15076 leader_election.cc:341] T 20a316e404414b288a315a5386208111 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:40.026674 15076 leader_election.cc:302] T 20a316e404414b288a315a5386208111 P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:40.027674 15980 raft_consensus.cc:2726] T 20a316e404414b288a315a5386208111 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 08:00:40.031931 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet a0887c4efd0f4c7c95e9d12734184a40 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0887c4efd0f4c7c95e9d12734184a40 already in progress: creating tablet
W20260810 08:00:40.034214 15076 leader_election.cc:341] T a92d01d83dbf4b6ba47464dcefa8b2ae P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
W20260810 08:00:40.042642 14678 catalog_manager.cc:4821] TS 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): delete failed for tablet fcd6d992683c4a5c886d7c5052da76b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fcd6d992683c4a5c886d7c5052da76b3 already in progress: creating tablet
I20260810 08:00:40.034735 15076 leader_election.cc:302] T a92d01d83dbf4b6ba47464dcefa8b2ae P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:40.050203 15980 raft_consensus.cc:2726] T a92d01d83dbf4b6ba47464dcefa8b2ae P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:40.054579 15890 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34630 (request call id 160) took 5149 ms. Trace:
I20260810 08:00:39.921366 15958 raft_consensus.cc:504] T e13b91d61063446f9d1454da4f8bb706 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:40.057251 15890 rpcz_store.cc:276] 0810 08:00:34.905598 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:34.995810 (+ 90212us) service_pool.cc:227] Handling call
0810 08:00:40.009572 (+5013762us) ts_tablet_manager.cc:1145] Deleting tablet 521d16ada2424d3aa41f838a511131fb
0810 08:00:40.054200 (+ 44628us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:40.054514 (+ 314us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4932868}
W20260810 08:00:40.065145 14686 catalog_manager.cc:6530] Service unavailable: an error occurred while writing to the sys-catalog: leader is not yet ready
I20260810 08:00:40.060639 15145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13b91d61063446f9d1454da4f8bb706" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07"
I20260810 08:00:40.065987 14686 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:42300 (request call id 262) took 1897 ms. Trace:
I20260810 08:00:40.066414 15145 raft_consensus.cc:3037] T e13b91d61063446f9d1454da4f8bb706 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:40.066303 14686 rpcz_store.cc:276] 0810 08:00:38.168733 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:39.179423 (+1010690us) service_pool.cc:227] Handling call
0810 08:00:39.224895 (+ 45472us) spinlock_profiling.cc:243] Waited 41.6 ms on lock 0x7b5400021554. stack: 00007fe2372c4033 00007fe2372c3f30 00007fe236e45534 00000000004db2b4 00000000004db249 00007fe23e1d0af0 00007fe23e253f2a 00007fe24011fd6c 00007fe24021db81 00007fe23dd78d37 00007fe23dd78cc2 00007fe23dd78bf1 00007fe23dd78b6c 00007fe23dd77e22 00007fe23bffa94c 00007fe23bff9d86
0810 08:00:40.064765 (+839870us) write_op.cc:489] Releasing partition, row and schema locks
0810 08:00:40.064788 (+ 23us) write_op.cc:454] Released schema lock
0810 08:00:40.065919 (+ 1131us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 08:00:40.064557 (+ 0us) op_driver.cc:372] HandleFailure(Service unavailable: leader is not yet ready)
0810 08:00:40.064597 (+ 40us) write_op.cc:489] Releasing partition, row and schema locks
0810 08:00:40.064619 (+ 22us) write_op.cc:454] Released schema lock
0810 08:00:40.064642 (+ 23us) write_op.cc:335] FINISH: Op aborted
Metrics: {"spinlock_wait_cycles":95623040,"child_traces":[["op",{}]]}
F20260810 08:00:40.068517 15385 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Service unavailable: an error occurred while writing to the sys-catalog: leader is not yet ready
*** Check failure stack trace: ***
*** Aborted at 1786348840 (unix time) try "date -d @1786348840" if you are using GNU date ***
I20260810 08:00:40.076679 15145 raft_consensus.cc:2454] T e13b91d61063446f9d1454da4f8bb706 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 1.
I20260810 08:00:40.079515 16056 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [term 4 LEADER]: Becoming Leader. State: Replica: 38a7164890624423a7427871aa2b10e0, State: Running, Role: LEADER
I20260810 08:00:40.080657 16056 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 286, Committed index: 286, Last appended: 2.289, Last appended by leader: 289, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:39.997872 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461de46896ee492b9f7cd1202208c19a" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
W20260810 08:00:40.005683 14918 leader_election.cc:341] T e13b91d61063446f9d1454da4f8bb706 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:40.078845 14922 leader_election.cc:302] T e13b91d61063446f9d1454da4f8bb706 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224; no voters:
W20260810 08:00:40.086756 14918 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.195:37729, user_credentials={real_user=slave}} blocked reactor thread for 81145.9us
I20260810 08:00:40.086879 16070 raft_consensus.cc:2781] T e13b91d61063446f9d1454da4f8bb706 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader election won for term 1
W20260810 08:00:40.089749 15079 leader_election.cc:341] T 461de46896ee492b9f7cd1202208c19a P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:40.090287 15079 leader_election.cc:302] T 461de46896ee492b9f7cd1202208c19a P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:39.970130 15437 tablet_bootstrap.cc:492] T 8e603d04f7454501b2edfb5a1f4ea7f0 P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:40.093063 15437 ts_tablet_manager.cc:1397] T 8e603d04f7454501b2edfb5a1f4ea7f0 P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.287s user 0.012s sys 0.003s
I20260810 08:00:40.093173 15980 raft_consensus.cc:2726] T 461de46896ee492b9f7cd1202208c19a P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:40.077329 16054 consensus_queue.cc:1225] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [LEADER]: Peer 38a7164890624423a7427871aa2b10e0 log is divergent from this leader: its last log entry 2.289 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 286
I20260810 08:00:40.094198 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13b91d61063446f9d1454da4f8bb706" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae"
I20260810 08:00:40.095397 16099 raft_consensus.cc:3032] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 3 LEADER]: Stepping down as leader of term 3
I20260810 08:00:40.095777 16099 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 99f2d868d76c440f8443b33e1cd19f58, State: Running, Role: LEADER
I20260810 08:00:40.096812 16099 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 286, Committed index: 286, Last appended: 3.287, Last appended by leader: 287, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:40.099395 16099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 3 FOLLOWER]: Advancing to term 4
I20260810 08:00:40.101789 16055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38a7164890624423a7427871aa2b10e0. Latest consensus state: current_term: 4 leader_uuid: "38a7164890624423a7427871aa2b10e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } } }
I20260810 08:00:40.102656 16055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [sys.catalog]: This master's current role is: LEADER
I20260810 08:00:40.112742 15890 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34630 (request call id 169) took 4187 ms. Trace:
I20260810 08:00:40.126760 15980 raft_consensus.cc:482] T 3855a2b4d90a47ad918c7a90cc8c7e4a P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:40.127146 14690 catalog_manager.cc:5587] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265) does not have the latest schema for tablet a4a78b2a960d4d78aa1f48a9c9edf3be (table default.Table_89fe7ad723964551909b73ff6b72b9af [id=ce96e7fb7c46404a859ca08c1fd1b26a]). Expected version 1 got 0
I20260810 08:00:40.127235 15980 raft_consensus.cc:504] T 3855a2b4d90a47ad918c7a90cc8c7e4a P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:40.128845 14690 catalog_manager.cc:5587] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265) does not have the latest schema for tablet 0687d323788642f798465ece973702bb (table default.Table_c65aa418eb904b57aab18f8b403e8c8e [id=0d60b763c4804818ae26fa693a4fd808]). Expected version 1 got 0
I20260810 08:00:40.129132 15980 leader_election.cc:288] T 3855a2b4d90a47ad918c7a90cc8c7e4a P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:40.130043 14690 catalog_manager.cc:5587] TS 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265) does not have the latest schema for tablet ae51ddeb22fe4cf7a8aa2d8b2b70a535 (table default.Table_1ed39e9362eb43ca8b58ae159fdd59d8 [id=df7590268b84411d983967b1b05d419f]). Expected version 1 got 0
I20260810 08:00:40.115917 15490 tablet_bootstrap.cc:654] T 8b5a629a7f8041179a006fa412fb6b48 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
W20260810 08:00:40.131723 14687 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0687d323788642f798465ece973702bb consensus state has no leader
I20260810 08:00:40.132478 14687 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:42300 (request call id 260) took 1980 ms. Trace:
I20260810 08:00:40.133522 15890 rpcz_store.cc:276] 0810 08:00:35.925156 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:35.955178 (+ 30022us) service_pool.cc:227] Handling call
0810 08:00:40.112325 (+4157147us) ts_tablet_manager.cc:1145] Deleting tablet fc0169c909454088b90ae8a5e01c1ec6
0810 08:00:40.112373 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:40.112675 (+ 302us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4147547}
I20260810 08:00:40.136384 15890 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34630 (request call id 168) took 4211 ms. Trace:
I20260810 08:00:40.132773 14687 rpcz_store.cc:276] 0810 08:00:38.152149 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:38.271911 (+119762us) service_pool.cc:227] Handling call
0810 08:00:38.272882 (+ 971us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 08:00:38.273569 (+ 687us) catalog_manager.cc:3616] Apply alter schema
0810 08:00:38.294334 (+ 20765us) catalog_manager.cc:3639] Acquired catalog manager lock
0810 08:00:38.294589 (+ 255us) catalog_manager.cc:3786] Updating metadata on disk
0810 08:00:39.187858 (+893269us) catalog_manager.cc:3817] Committing alterations to in-memory state
0810 08:00:40.132389 (+944531us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 08:00:38.889253 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 08:00:38.889583 (+ 330us) write_op.cc:432] Acquiring schema lock in shared mode
0810 08:00:38.889604 (+ 21us) write_op.cc:435] Acquired schema lock
0810 08:00:38.889638 (+ 34us) tablet.cc:660] Decoding operations
0810 08:00:38.889866 (+ 228us) write_op.cc:620] Acquiring the partition lock for write op
0810 08:00:38.889897 (+ 31us) write_op.cc:641] Partition lock acquired for write op
0810 08:00:38.889918 (+ 21us) tablet.cc:683] Acquiring locks for 1 operations
0810 08:00:38.890039 (+ 121us) tablet.cc:699] Row locks acquired
0810 08:00:38.890052 (+ 13us) write_op.cc:260] PREPARE: finished
0810 08:00:38.890144 (+ 92us) write_op.cc:270] Start()
0810 08:00:38.890214 (+ 70us) write_op.cc:276] Timestamp: P: 1786348838890099 usec, L: 0
0810 08:00:38.890229 (+ 15us) op_driver.cc:331] REPLICATION: starting
0810 08:00:38.890747 (+ 518us) log.cc:842] Serialized 405 byte log entry
0810 08:00:39.156922 (+266175us) op_driver.cc:444] REPLICATION: finished
0810 08:00:39.183596 (+ 26674us) write_op.cc:301] APPLY: starting
0810 08:00:39.183685 (+ 89us) tablet.cc:1365] starting BulkCheckPresence
0810 08:00:39.183934 (+ 249us) tablet.cc:1368] finished BulkCheckPresence
0810 08:00:39.184072 (+ 138us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 08:00:39.184366 (+ 294us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 08:00:39.184725 (+ 359us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 08:00:39.184784 (+ 59us) write_op.cc:312] APPLY: finished
0810 08:00:39.187263 (+ 2479us) log.cc:842] Serialized 23 byte log entry
0810 08:00:39.187513 (+ 250us) write_op.cc:489] Releasing partition, row and schema locks
0810 08:00:39.187575 (+ 62us) write_op.cc:454] Released schema lock
0810 08:00:39.187619 (+ 44us) write_op.cc:341] FINISH: Updating metrics
0810 08:00:39.188183 (+ 564us) write_op.cc:489] Releasing partition, row and schema locks
0810 08:00:39.188240 (+ 57us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":26418,"apply.run_cpu_time_us":2448,"apply.run_wall_time_us":4936,"num_ops":1,"prepare.queue_time_us":593413,"prepare.run_cpu_time_us":1762,"prepare.run_wall_time_us":150122,"replication_time_us":266630}]]}
I20260810 08:00:40.146062 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3855a2b4d90a47ad918c7a90cc8c7e4a" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
W20260810 08:00:40.148538 15076 leader_election.cc:341] T 3855a2b4d90a47ad918c7a90cc8c7e4a P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
E20260810 08:00:40.153041 14861 catalog_manager.cc:816] Error processing pending assignments: Service unavailable: error persisting updated tablet metadata: leader is not yet ready
W20260810 08:00:40.155478 14681 catalog_manager.cc:4821] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet 521d16ada2424d3aa41f838a511131fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 521d16ada2424d3aa41f838a511131fb already in progress: creating tablet
I20260810 08:00:40.156157 15439 tablet_bootstrap.cc:492] T adada21bc1bf4f19a6347d0935bd74bd P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
W20260810 08:00:40.156814 14681 catalog_manager.cc:4821] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet fc0169c909454088b90ae8a5e01c1ec6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc0169c909454088b90ae8a5e01c1ec6 already in progress: creating tablet
PC: @ 0x0 (unknown)
I20260810 08:00:40.166301 15439 ts_tablet_manager.cc:1397] T adada21bc1bf4f19a6347d0935bd74bd P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.430s user 0.019s sys 0.002s
W20260810 08:00:40.167243 14678 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 4ca688b4031c41a9884829fbd0be1f33 (table default.Table_8dd0e3d12b2c461dbedf3a2ac454cf93 [id=da2bb7febfe9421380f61df1393ea3b0], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4ca688b4031c41a9884829fbd0be1f33 consensus state has no leader
I20260810 08:00:40.130472 16070 raft_consensus.cc:686] T e13b91d61063446f9d1454da4f8bb706 P a91b27762182454baada71e91ceab224 [term 1 LEADER]: Becoming Leader. State: Replica: a91b27762182454baada71e91ceab224, State: Running, Role: LEADER
I20260810 08:00:40.140452 15958 leader_election.cc:288] T e13b91d61063446f9d1454da4f8bb706 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Requested vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:40.118701 16103 raft_consensus.cc:482] T 3ac554bd37b045a2a8cf2743f4645f28 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:40.158438 15437 raft_consensus.cc:348] T 8e603d04f7454501b2edfb5a1f4ea7f0 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:40.171074 15437 raft_consensus.cc:374] T 8e603d04f7454501b2edfb5a1f4ea7f0 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:40.170658 16070 consensus_queue.cc:237] T e13b91d61063446f9d1454da4f8bb706 P a91b27762182454baada71e91ceab224 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:40.171325 15437 raft_consensus.cc:729] T 8e603d04f7454501b2edfb5a1f4ea7f0 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
W20260810 08:00:40.161821 14918 leader_election.cc:341] T e13b91d61063446f9d1454da4f8bb706 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:40.176043 15890 rpcz_store.cc:276] 0810 08:00:35.924788 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:35.954795 (+ 30007us) service_pool.cc:227] Handling call
0810 08:00:40.135895 (+4181100us) ts_tablet_manager.cc:1145] Deleting tablet abc607ab01f74b2c9c693d6dea73090c
0810 08:00:40.135945 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:40.136327 (+ 382us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2945,"tablet-delete.queue_time_us":4156334}
*** SIGABRT (@0x3e8000038e7) received by PID 14567 (TID 0x7f3ee23fd700) from PID 14567; stack trace: ***
I20260810 08:00:40.186483 14567 master-stress-test.cc:578] Killing test thread 1/10
I20260810 08:00:40.177584 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3855a2b4d90a47ad918c7a90cc8c7e4a" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
W20260810 08:00:40.191385 15079 leader_election.cc:341] T 3855a2b4d90a47ad918c7a90cc8c7e4a P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:40.191784 15079 leader_election.cc:302] T 3855a2b4d90a47ad918c7a90cc8c7e4a P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:40.178573 15437 consensus_queue.cc:260] T 8e603d04f7454501b2edfb5a1f4ea7f0 P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:40.170454 16103 raft_consensus.cc:504] T 3ac554bd37b045a2a8cf2743f4645f28 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:40.193552 15797 raft_consensus.cc:1226] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [term 4 FOLLOWER]: Rejecting Update request from peer 99f2d868d76c440f8443b33e1cd19f58 for earlier term 3. Current term is 4. Ops: [2.182-3.287]
I20260810 08:00:40.195899 16103 leader_election.cc:288] T 3ac554bd37b045a2a8cf2743f4645f28 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:40.197738 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ac554bd37b045a2a8cf2743f4645f28" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
W20260810 08:00:40.199250 14918 leader_election.cc:341] T 3ac554bd37b045a2a8cf2743f4645f28 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
W20260810 08:00:40.200309 14681 catalog_manager.cc:4821] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet abc607ab01f74b2c9c693d6dea73090c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abc607ab01f74b2c9c693d6dea73090c already in progress: creating tablet
I20260810 08:00:40.201534 15980 raft_consensus.cc:2726] T 3855a2b4d90a47ad918c7a90cc8c7e4a P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:40.203339 15890 tablet_replica.cc:330] T 8364045e3fa2441e8dc1a1d87f73361d P a91b27762182454baada71e91ceab224: stopping tablet replica
W20260810 08:00:40.203428 14922 leader_election.cc:341] T 3ac554bd37b045a2a8cf2743f4645f28 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Not found: Tablet not found: 3ac554bd37b045a2a8cf2743f4645f28
I20260810 08:00:40.203998 14922 leader_election.cc:302] T 3ac554bd37b045a2a8cf2743f4645f28 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a91b27762182454baada71e91ceab224; no voters: 3fc5e568f3724f158b616c303efb9eae, 9d7219686f304f62a54acc7497cbbd07
I20260810 08:00:40.205044 16102 raft_consensus.cc:2726] T 3ac554bd37b045a2a8cf2743f4645f28 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:40.204031 15437 ts_tablet_manager.cc:1428] T 8e603d04f7454501b2edfb5a1f4ea7f0 P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.057s user 0.005s sys 0.001s
I20260810 08:00:40.209187 15437 tablet_bootstrap.cc:492] T adada21bc1bf4f19a6347d0935bd74bd P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:40.211670 15490 tablet_bootstrap.cc:492] T 8b5a629a7f8041179a006fa412fb6b48 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:40.212128 15490 ts_tablet_manager.cc:1397] T 8b5a629a7f8041179a006fa412fb6b48 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.266s user 0.009s sys 0.002s
I20260810 08:00:40.214330 15890 raft_consensus.cc:2229] T 8364045e3fa2441e8dc1a1d87f73361d P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus shutting down.
@ 0x2b0af0 __tsan::CallUserSignalHandler() at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2b31e0 rtl_sigaction() at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
I20260810 08:00:40.254300 15890 raft_consensus.cc:2258] T 8364045e3fa2441e8dc1a1d87f73361d P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 08:00:40.254118 15797 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [term 4 FOLLOWER]: Refusing update from remote peer 38a7164890624423a7427871aa2b10e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 181. Preceding OpId from leader: term: 4 index: 291. (index mismatch)
I20260810 08:00:40.256767 16055 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 38a7164890624423a7427871aa2b10e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 290, Last known committed idx: 180, Time since last communication: 0.001s
I20260810 08:00:40.240653 16102 raft_consensus.cc:482] T 389b30cba2b149e9ab0de8c191808db2 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:40.260708 16102 raft_consensus.cc:504] T 389b30cba2b149e9ab0de8c191808db2 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:40.263875 16089 catalog_manager.cc:669] Loaded metadata for table default.Table_d48d3754e2e84c28b58869ee05f4ea95 [id=0ae329e15cfe4dc0a3e0de17663b058d]
I20260810 08:00:40.265547 15145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "389b30cba2b149e9ab0de8c191808db2" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:40.266425 15145 raft_consensus.cc:2454] T 389b30cba2b149e9ab0de8c191808db2 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 0.
I20260810 08:00:40.267930 14922 leader_election.cc:302] T 389b30cba2b149e9ab0de8c191808db2 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224; no voters:
I20260810 08:00:40.268680 16070 raft_consensus.cc:2781] T 389b30cba2b149e9ab0de8c191808db2 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:40.268847 16102 leader_election.cc:288] T 389b30cba2b149e9ab0de8c191808db2 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:40.269037 16070 raft_consensus.cc:482] T 389b30cba2b149e9ab0de8c191808db2 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:40.269564 16070 raft_consensus.cc:3037] T 389b30cba2b149e9ab0de8c191808db2 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:40.271510 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "389b30cba2b149e9ab0de8c191808db2" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
W20260810 08:00:40.273042 14918 leader_election.cc:341] T 389b30cba2b149e9ab0de8c191808db2 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:40.169443 15439 raft_consensus.cc:348] T adada21bc1bf4f19a6347d0935bd74bd P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:40.277189 15439 raft_consensus.cc:374] T adada21bc1bf4f19a6347d0935bd74bd P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:40.278280 15437 tablet_bootstrap.cc:654] T adada21bc1bf4f19a6347d0935bd74bd P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:40.280011 15490 raft_consensus.cc:348] T 8b5a629a7f8041179a006fa412fb6b48 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:40.280689 15490 raft_consensus.cc:374] T 8b5a629a7f8041179a006fa412fb6b48 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:40.280987 15490 raft_consensus.cc:729] T 8b5a629a7f8041179a006fa412fb6b48 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:40.281720 15490 consensus_queue.cc:260] T 8b5a629a7f8041179a006fa412fb6b48 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:40.282737 15890 ts_tablet_manager.cc:1905] T 8364045e3fa2441e8dc1a1d87f73361d P a91b27762182454baada71e91ceab224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 08:00:40.277483 15439 raft_consensus.cc:729] T adada21bc1bf4f19a6347d0935bd74bd P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:40.290359 15439 consensus_queue.cc:260] T adada21bc1bf4f19a6347d0935bd74bd P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
@ 0x7f3ef70dc980 (unknown) at ??:0
I20260810 08:00:40.327144 15439 ts_tablet_manager.cc:1428] T adada21bc1bf4f19a6347d0935bd74bd P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.160s user 0.002s sys 0.003s
I20260810 08:00:40.265555 16089 catalog_manager.cc:669] Loaded metadata for table default.Table_4ff94124f5cb4710a0ad3ff1781efcd3 [id=0b1956596de94b569ebd9f1816e4b699]
W20260810 08:00:40.351033 14680 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.14.57.254:43757 timed out after 0.563s (SENT)
I20260810 08:00:40.365090 15439 tablet_bootstrap.cc:492] T e13b91d61063446f9d1454da4f8bb706 P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:40.371729 16103 raft_consensus.cc:482] T 8e603d04f7454501b2edfb5a1f4ea7f0 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:40.372251 16103 raft_consensus.cc:504] T 8e603d04f7454501b2edfb5a1f4ea7f0 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:40.374547 16103 leader_election.cc:288] T 8e603d04f7454501b2edfb5a1f4ea7f0 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:40.376458 15439 tablet_bootstrap.cc:654] T e13b91d61063446f9d1454da4f8bb706 P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:40.278894 16070 raft_consensus.cc:504] T 389b30cba2b149e9ab0de8c191808db2 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
W20260810 08:00:40.380550 14922 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.14.57.194:36265, user_credentials={real_user=slave}} blocked reactor thread for 88563.7us
I20260810 08:00:40.382922 15145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e603d04f7454501b2edfb5a1f4ea7f0" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07" is_pre_election: true
I20260810 08:00:40.383541 15145 raft_consensus.cc:2454] T 8e603d04f7454501b2edfb5a1f4ea7f0 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 0.
I20260810 08:00:40.384595 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e603d04f7454501b2edfb5a1f4ea7f0" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
I20260810 08:00:40.383061 16089 catalog_manager.cc:669] Loaded metadata for table default.Table_d734cf52bb724d51802568793b928421 [id=0d60b763c4804818ae26fa693a4fd808]
I20260810 08:00:40.385247 15490 ts_tablet_manager.cc:1428] T 8b5a629a7f8041179a006fa412fb6b48 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.173s user 0.005s sys 0.005s
W20260810 08:00:40.393599 14918 leader_election.cc:341] T 8e603d04f7454501b2edfb5a1f4ea7f0 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:40.403256 14922 leader_election.cc:302] T 8e603d04f7454501b2edfb5a1f4ea7f0 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224; no voters: 3fc5e568f3724f158b616c303efb9eae
I20260810 08:00:40.404026 16102 raft_consensus.cc:2781] T 8e603d04f7454501b2edfb5a1f4ea7f0 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:40.404341 16102 raft_consensus.cc:482] T 8e603d04f7454501b2edfb5a1f4ea7f0 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:40.404639 16102 raft_consensus.cc:3037] T 8e603d04f7454501b2edfb5a1f4ea7f0 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:40.418565 16089 catalog_manager.cc:669] Loaded metadata for table default.Table_e0d4352887e04b38939d453e07a37ef9 [id=11ca1b81b7704d33820eef06f5a25310]
I20260810 08:00:40.419214 16059 raft_consensus.cc:482] T 579bac622ddf40bf80d01d66ae2fa100 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:40.419842 16059 raft_consensus.cc:504] T 579bac622ddf40bf80d01d66ae2fa100 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:40.421912 15490 tablet_bootstrap.cc:492] T 3090226feb724bb1aecb0f559d8282a8 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:40.424264 16059 leader_election.cc:288] T 579bac622ddf40bf80d01d66ae2fa100 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:40.425899 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "579bac622ddf40bf80d01d66ae2fa100" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:40.426318 16070 leader_election.cc:288] T 389b30cba2b149e9ab0de8c191808db2 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Requested vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
@ 0x7f3ef2399e87 gsignal at ??:0
W20260810 08:00:40.428442 15234 leader_election.cc:341] T 579bac622ddf40bf80d01d66ae2fa100 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:40.429980 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "389b30cba2b149e9ab0de8c191808db2" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae"
W20260810 08:00:40.432761 15234 leader_election.cc:341] T 579bac622ddf40bf80d01d66ae2fa100 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Not found: Tablet not found: 579bac622ddf40bf80d01d66ae2fa100
W20260810 08:00:40.433992 14918 leader_election.cc:341] T 389b30cba2b149e9ab0de8c191808db2 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:40.433233 15234 leader_election.cc:302] T 579bac622ddf40bf80d01d66ae2fa100 P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae; no voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224
I20260810 08:00:40.436990 15145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "389b30cba2b149e9ab0de8c191808db2" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07"
I20260810 08:00:40.438434 16059 raft_consensus.cc:2726] T 579bac622ddf40bf80d01d66ae2fa100 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:40.437885 15145 raft_consensus.cc:3037] T 389b30cba2b149e9ab0de8c191808db2 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:40.444649 16089 catalog_manager.cc:669] Loaded metadata for table default.Table_d423114f2dce44ffaf0fad0c40c5b985 [id=1ea3fa32feba46c796a5f7ab529516f6]
W20260810 08:00:40.459690 14680 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.14.57.254:43757, user_credentials={real_user=slave}} blocked reactor thread for 108782us
@ 0x7f3ef239b7f1 abort at ??:0
@ 0x2b4f37 __interceptor_abort at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
I20260810 08:00:40.464725 16102 raft_consensus.cc:504] T 8e603d04f7454501b2edfb5a1f4ea7f0 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:40.497277 15437 tablet_bootstrap.cc:492] T adada21bc1bf4f19a6347d0935bd74bd P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:40.499737 15437 ts_tablet_manager.cc:1397] T adada21bc1bf4f19a6347d0935bd74bd P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.291s user 0.015s sys 0.004s
I20260810 08:00:40.479846 16089 catalog_manager.cc:669] Loaded metadata for table default.Table_e3bf1bdea6324b319c1a04b7231674f5 [id=20c0d637b1c34aff9e999f35d840f169]
I20260810 08:00:40.521716 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e603d04f7454501b2edfb5a1f4ea7f0" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae"
W20260810 08:00:40.523506 14918 leader_election.cc:341] T 8e603d04f7454501b2edfb5a1f4ea7f0 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:40.530501 15490 tablet_bootstrap.cc:654] T 3090226feb724bb1aecb0f559d8282a8 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:40.530790 16070 consensus_queue.cc:1035] T e0652a7684b04a89840049ce0d781199 P a91b27762182454baada71e91ceab224 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 08:00:40.506475 15437 raft_consensus.cc:348] T adada21bc1bf4f19a6347d0935bd74bd P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:40.536355 15437 raft_consensus.cc:374] T adada21bc1bf4f19a6347d0935bd74bd P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:40.536621 15437 raft_consensus.cc:729] T adada21bc1bf4f19a6347d0935bd74bd P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:40.537215 15437 consensus_queue.cc:260] T adada21bc1bf4f19a6347d0935bd74bd P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:40.537665 15146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e603d04f7454501b2edfb5a1f4ea7f0" candidate_uuid: "a91b27762182454baada71e91ceab224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7219686f304f62a54acc7497cbbd07"
I20260810 08:00:40.538434 15146 raft_consensus.cc:3037] T 8e603d04f7454501b2edfb5a1f4ea7f0 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Advancing to term 1
@ 0x7f3ef53cddf4 google::LogMessage::SendToLog() at ??:0
I20260810 08:00:40.546648 15146 raft_consensus.cc:2454] T 8e603d04f7454501b2edfb5a1f4ea7f0 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 1.
I20260810 08:00:40.548280 14922 leader_election.cc:302] T 8e603d04f7454501b2edfb5a1f4ea7f0 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224; no voters: 3fc5e568f3724f158b616c303efb9eae
I20260810 08:00:40.549058 15437 ts_tablet_manager.cc:1428] T adada21bc1bf4f19a6347d0935bd74bd P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.049s user 0.007s sys 0.000s
I20260810 08:00:40.549606 16070 raft_consensus.cc:2781] T 8e603d04f7454501b2edfb5a1f4ea7f0 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 08:00:40.550065 16070 raft_consensus.cc:686] T 8e603d04f7454501b2edfb5a1f4ea7f0 P a91b27762182454baada71e91ceab224 [term 1 LEADER]: Becoming Leader. State: Replica: a91b27762182454baada71e91ceab224, State: Running, Role: LEADER
I20260810 08:00:40.550820 16070 consensus_queue.cc:237] T 8e603d04f7454501b2edfb5a1f4ea7f0 P a91b27762182454baada71e91ceab224 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:40.550092 15437 tablet_bootstrap.cc:492] T 8a8454f693494ab3b82f5b85c2d62bb2 P a91b27762182454baada71e91ceab224: Bootstrap starting.
I20260810 08:00:40.552773 15890 ts_tablet_manager.cc:1918] T 8364045e3fa2441e8dc1a1d87f73361d P a91b27762182454baada71e91ceab224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 08:00:40.553151 15890 log.cc:1196] T 8364045e3fa2441e8dc1a1d87f73361d P a91b27762182454baada71e91ceab224: Deleting WAL directory at /tmp/dist-test-taskzwAODw/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786348806871146-14567-0/minicluster-data/ts-0/wal/wals/8364045e3fa2441e8dc1a1d87f73361d
I20260810 08:00:40.519697 16089 catalog_manager.cc:669] Loaded metadata for table default.Table_fbd8cd0e547e4fdd8413c51f0f7da212 [id=22b1e5a875e142ecb7fe1ba06f144e77]
I20260810 08:00:40.554965 15490 tablet_bootstrap.cc:492] T 3090226feb724bb1aecb0f559d8282a8 P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:40.555336 15490 ts_tablet_manager.cc:1397] T 3090226feb724bb1aecb0f559d8282a8 P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.134s user 0.009s sys 0.003s
I20260810 08:00:40.559484 15437 tablet_bootstrap.cc:654] T 8a8454f693494ab3b82f5b85c2d62bb2 P a91b27762182454baada71e91ceab224: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:40.557510 15490 raft_consensus.cc:348] T 3090226feb724bb1aecb0f559d8282a8 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:40.570252 15490 raft_consensus.cc:374] T 3090226feb724bb1aecb0f559d8282a8 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:40.570523 15490 raft_consensus.cc:729] T 3090226feb724bb1aecb0f559d8282a8 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:40.571089 15490 consensus_queue.cc:260] T 3090226feb724bb1aecb0f559d8282a8 P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:40.572952 15490 ts_tablet_manager.cc:1428] T 3090226feb724bb1aecb0f559d8282a8 P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.017s user 0.004s sys 0.000s
I20260810 08:00:40.601085 16089 catalog_manager.cc:669] Loaded metadata for table default.Table_9a8eb842ebf244d5abbfb0f43373ca08 [id=23114ccea26d48079f910570c9580a71]
I20260810 08:00:40.602777 16089 catalog_manager.cc:669] Loaded metadata for table default.Table_ccb04058c52049ba833379f6cdbb9df2 [id=265b23679d844c249d2c2e663a3671dc]
I20260810 08:00:40.604820 15890 ts_tablet_manager.cc:1939] T 8364045e3fa2441e8dc1a1d87f73361d P a91b27762182454baada71e91ceab224: Deleting consensus metadata
I20260810 08:00:40.614318 15490 tablet_bootstrap.cc:492] T 3593a8e38e1a44f68a54bd52b19a3c6d P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
I20260810 08:00:40.617967 15980 raft_consensus.cc:482] T abc607ab01f74b2c9c693d6dea73090c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:40.618525 15980 raft_consensus.cc:504] T abc607ab01f74b2c9c693d6dea73090c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:40.620995 15980 leader_election.cc:288] T abc607ab01f74b2c9c693d6dea73090c P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:40.621711 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abc607ab01f74b2c9c693d6dea73090c" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5e568f3724f158b616c303efb9eae" is_pre_election: true
W20260810 08:00:40.622880 15076 leader_election.cc:341] T abc607ab01f74b2c9c693d6dea73090c P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:40.623751 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abc607ab01f74b2c9c693d6dea73090c" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
W20260810 08:00:40.626359 15079 leader_election.cc:341] T abc607ab01f74b2c9c693d6dea73090c P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Illegal state: must be running to vote when last-logged opid is not known
I20260810 08:00:40.626698 15079 leader_election.cc:302] T abc607ab01f74b2c9c693d6dea73090c P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07; no voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224
I20260810 08:00:40.627552 15980 raft_consensus.cc:2726] T abc607ab01f74b2c9c693d6dea73090c P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 08:00:40.631103 16099 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:40.631904 15490 tablet_bootstrap.cc:654] T 3593a8e38e1a44f68a54bd52b19a3c6d P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:40.631695 16099 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
I20260810 08:00:40.634931 14705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99f2d868d76c440f8443b33e1cd19f58" candidate_term: 5 candidate_status { last_received { term: 3 index: 287 } } ignore_live_leader: false dest_uuid: "38a7164890624423a7427871aa2b10e0" is_pre_election: true
I20260810 08:00:40.635317 16099 leader_election.cc:288] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 38a7164890624423a7427871aa2b10e0 (127.14.57.253:40439), 1e8cadb26fe0428a964f66ef8172f3be (127.14.57.252:33819)
I20260810 08:00:40.636441 16103 consensus_queue.cc:1035] T e0652a7684b04a89840049ce0d781199 P a91b27762182454baada71e91ceab224 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 08:00:40.636525 15796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99f2d868d76c440f8443b33e1cd19f58" candidate_term: 5 candidate_status { last_received { term: 3 index: 287 } } ignore_live_leader: false dest_uuid: "1e8cadb26fe0428a964f66ef8172f3be" is_pre_election: true
I20260810 08:00:40.638584 14597 leader_election.cc:302] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99f2d868d76c440f8443b33e1cd19f58; no voters: 1e8cadb26fe0428a964f66ef8172f3be, 38a7164890624423a7427871aa2b10e0
I20260810 08:00:40.639809 16099 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 99f2d868d76c440f8443b33e1cd19f58 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 08:00:40.647650 16089 catalog_manager.cc:669] Loaded metadata for table default.Table_4bed023fc139438b8753c1e1cf1dd469 [id=2ab9083d7ffb4485914939c2827547c6]
I20260810 08:00:40.609149 16102 leader_election.cc:288] T 8e603d04f7454501b2edfb5a1f4ea7f0 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Requested vote from peers 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:40.661940 15890 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34630 (request call id 172) took 4675 ms. Trace:
@ 0x7f3ef53ce9a0 google::LogMessage::Flush() at ??:0
I20260810 08:00:40.662609 15890 rpcz_store.cc:276] 0810 08:00:35.986270 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:35.989942 (+ 3672us) service_pool.cc:227] Handling call
0810 08:00:40.203164 (+4213222us) ts_tablet_manager.cc:1145] Deleting tablet 8364045e3fa2441e8dc1a1d87f73361d
0810 08:00:40.203215 (+ 51us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:40.424949 (+221734us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:40.522999 (+ 98050us) tablet_metadata.cc:641] Metadata flushed
0810 08:00:40.661857 (+138858us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10062,"tablet-delete.queue_time_us":4205885}
I20260810 08:00:40.664153 14681 catalog_manager.cc:4841] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): tablet 8364045e3fa2441e8dc1a1d87f73361d (table default.Table_290c3db6ca0e4fdf8e007f723b11d27d [id=97abcf4bf320496ebbec1fd2ed20765b]) successfully deleted
I20260810 08:00:40.670660 16089 catalog_manager.cc:669] Loaded metadata for table default.Table_73aba277eb504cc1917f096e97f4fd50 [id=2b9900d6a50549929028d26f1f1ccd4b]
I20260810 08:00:40.673236 16089 catalog_manager.cc:669] Loaded metadata for table default.Table_68ea7153b43948c19bb9a37d1cddc777 [id=2d69ae2042f24bbb856608ba8e500912]
I20260810 08:00:40.674073 15439 tablet_bootstrap.cc:492] T e13b91d61063446f9d1454da4f8bb706 P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:40.674513 15439 ts_tablet_manager.cc:1397] T e13b91d61063446f9d1454da4f8bb706 P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.310s user 0.039s sys 0.092s
I20260810 08:00:40.674831 16059 raft_consensus.cc:482] T adada21bc1bf4f19a6347d0935bd74bd P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:40.675288 16059 raft_consensus.cc:504] T adada21bc1bf4f19a6347d0935bd74bd P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:40.676682 15439 raft_consensus.cc:348] T e13b91d61063446f9d1454da4f8bb706 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:40.677129 16059 leader_election.cc:288] T adada21bc1bf4f19a6347d0935bd74bd P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265)
I20260810 08:00:40.677341 15439 raft_consensus.cc:374] T e13b91d61063446f9d1454da4f8bb706 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:40.677575 15439 raft_consensus.cc:729] T e13b91d61063446f9d1454da4f8bb706 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:40.678686 15439 consensus_queue.cc:260] T e13b91d61063446f9d1454da4f8bb706 P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
W20260810 08:00:40.679702 15234 leader_election.cc:341] T adada21bc1bf4f19a6347d0935bd74bd P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9d7219686f304f62a54acc7497cbbd07 (127.14.57.194:36265): Not found: Tablet not found: adada21bc1bf4f19a6347d0935bd74bd
I20260810 08:00:40.680661 15439 ts_tablet_manager.cc:1428] T e13b91d61063446f9d1454da4f8bb706 P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260810 08:00:40.681519 15439 tablet_bootstrap.cc:492] T 5990e7ea243843bb869de5dde3e6cd23 P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:40.686691 15890 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34630 (request call id 180) took 4281 ms. Trace:
I20260810 08:00:40.686585 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adada21bc1bf4f19a6347d0935bd74bd" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:40.687222 14987 raft_consensus.cc:2454] T adada21bc1bf4f19a6347d0935bd74bd P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 0.
I20260810 08:00:40.687464 15490 tablet_bootstrap.cc:492] T 3593a8e38e1a44f68a54bd52b19a3c6d P 9d7219686f304f62a54acc7497cbbd07: No bootstrap required, opened a new log
I20260810 08:00:40.687024 15890 rpcz_store.cc:276] 0810 08:00:36.405595 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:36.415519 (+ 9924us) service_pool.cc:227] Handling call
0810 08:00:40.664913 (+4249394us) ts_tablet_manager.cc:1145] Deleting tablet ed8f3e52959b4a0c8ed2a053e05e5bd6
0810 08:00:40.686328 (+ 21415us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:40.686629 (+ 301us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4248599}
I20260810 08:00:40.687851 15490 ts_tablet_manager.cc:1397] T 3593a8e38e1a44f68a54bd52b19a3c6d P 9d7219686f304f62a54acc7497cbbd07: Time spent bootstrapping tablet: real 0.074s user 0.008s sys 0.000s
W20260810 08:00:40.688338 14681 catalog_manager.cc:4821] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet ed8f3e52959b4a0c8ed2a053e05e5bd6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed8f3e52959b4a0c8ed2a053e05e5bd6 already in progress: creating tablet
I20260810 08:00:40.689481 15234 leader_election.cc:302] T adada21bc1bf4f19a6347d0935bd74bd P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224; no voters: 9d7219686f304f62a54acc7497cbbd07
I20260810 08:00:40.690685 16059 raft_consensus.cc:2781] T adada21bc1bf4f19a6347d0935bd74bd P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 08:00:40.691018 16059 raft_consensus.cc:482] T adada21bc1bf4f19a6347d0935bd74bd P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 08:00:40.691354 16059 raft_consensus.cc:3037] T adada21bc1bf4f19a6347d0935bd74bd P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Advancing to term 1
@ 0x7f3ef53d3adb google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x357eb6 kudu::MasterStressTest::ReplaceTabletThread() at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/master-stress-test.cc:?
I20260810 08:00:40.687361 15439 tablet_bootstrap.cc:654] T 5990e7ea243843bb869de5dde3e6cd23 P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:40.702947 16059 raft_consensus.cc:504] T adada21bc1bf4f19a6347d0935bd74bd P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:40.705093 16089 catalog_manager.cc:669] Loaded metadata for table default.Table_cb30530bbc01408888f095d0271f60c9 [id=2e3922de3fff4f028db003aa9fbdc789]
I20260810 08:00:40.706012 15439 tablet_bootstrap.cc:492] T 5990e7ea243843bb869de5dde3e6cd23 P 3fc5e568f3724f158b616c303efb9eae: No bootstrap required, opened a new log
I20260810 08:00:40.705935 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adada21bc1bf4f19a6347d0935bd74bd" candidate_uuid: "3fc5e568f3724f158b616c303efb9eae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224"
I20260810 08:00:40.706403 15439 ts_tablet_manager.cc:1397] T 5990e7ea243843bb869de5dde3e6cd23 P 3fc5e568f3724f158b616c303efb9eae: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.004s
I20260810 08:00:40.706318 15890 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34630 (request call id 183) took 4224 ms. Trace:
I20260810 08:00:40.706673 14987 raft_consensus.cc:3037] T adada21bc1bf4f19a6347d0935bd74bd P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Advancing to term 1
I20260810 08:00:40.709743 15145 raft_consensus.cc:2454] T 389b30cba2b149e9ab0de8c191808db2 P 9d7219686f304f62a54acc7497cbbd07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91b27762182454baada71e91ceab224 in term 1.
I20260810 08:00:40.711040 14922 leader_election.cc:302] T 389b30cba2b149e9ab0de8c191808db2 P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d7219686f304f62a54acc7497cbbd07, a91b27762182454baada71e91ceab224; no voters: 3fc5e568f3724f158b616c303efb9eae
I20260810 08:00:40.711752 15490 raft_consensus.cc:348] T 3593a8e38e1a44f68a54bd52b19a3c6d P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:40.712415 15490 raft_consensus.cc:374] T 3593a8e38e1a44f68a54bd52b19a3c6d P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:40.712654 15490 raft_consensus.cc:729] T 3593a8e38e1a44f68a54bd52b19a3c6d P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7219686f304f62a54acc7497cbbd07, State: Initialized, Role: FOLLOWER
I20260810 08:00:40.713095 16070 raft_consensus.cc:2781] T 389b30cba2b149e9ab0de8c191808db2 P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 08:00:40.713739 16070 raft_consensus.cc:686] T 389b30cba2b149e9ab0de8c191808db2 P a91b27762182454baada71e91ceab224 [term 1 LEADER]: Becoming Leader. State: Replica: a91b27762182454baada71e91ceab224, State: Running, Role: LEADER
I20260810 08:00:40.714268 14987 raft_consensus.cc:2454] T adada21bc1bf4f19a6347d0935bd74bd P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fc5e568f3724f158b616c303efb9eae in term 1.
I20260810 08:00:40.714550 16070 consensus_queue.cc:237] T 389b30cba2b149e9ab0de8c191808db2 P a91b27762182454baada71e91ceab224 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:40.715652 16089 catalog_manager.cc:669] Loaded metadata for table default.Table_812e1f2d005b4c1a8edc57a1c48b7fa0 [id=2f9a7edf6d114777b7444e8d6b03588b]
I20260810 08:00:40.717186 16089 catalog_manager.cc:669] Loaded metadata for table default.Table_ca56170c42664687b01194771c981e30 [id=321031ecbd1142a69bdef6900e1c15de]
I20260810 08:00:40.717931 15234 leader_election.cc:302] T adada21bc1bf4f19a6347d0935bd74bd P 3fc5e568f3724f158b616c303efb9eae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fc5e568f3724f158b616c303efb9eae, a91b27762182454baada71e91ceab224; no voters:
I20260810 08:00:40.708655 15439 raft_consensus.cc:348] T 5990e7ea243843bb869de5dde3e6cd23 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:40.718674 16089 catalog_manager.cc:669] Loaded metadata for table default.Table_ac97594854254f2a9beb18b682277e3d [id=350692a478034b5f97629024f9ec072e]
I20260810 08:00:40.718884 15439 raft_consensus.cc:374] T 5990e7ea243843bb869de5dde3e6cd23 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:40.719134 15439 raft_consensus.cc:729] T 5990e7ea243843bb869de5dde3e6cd23 P 3fc5e568f3724f158b616c303efb9eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Initialized, Role: FOLLOWER
I20260810 08:00:40.719842 16127 raft_consensus.cc:2781] T adada21bc1bf4f19a6347d0935bd74bd P 3fc5e568f3724f158b616c303efb9eae [term 1 FOLLOWER]: Leader election won for term 1
I20260810 08:00:40.720038 16089 catalog_manager.cc:669] Loaded metadata for table default.Table_7893219bf6754edbba16135d2605adce [id=3afd9e56a6fb4bf5bec58c10b7883f2e]
I20260810 08:00:40.719754 15439 consensus_queue.cc:260] T 5990e7ea243843bb869de5dde3e6cd23 P 3fc5e568f3724f158b616c303efb9eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
@ 0x3874ce _ZNSt3__18__invokeIMN4kudu16MasterStressTestEFvvEPNS1_26MasterStressTest_Test_TestEJEvEEDTcldsdeclsr3std3__1E7forwardIT0_Efp0_Efp_spclsr3std3__1E7forwardIT1_Efp1_EEEOT_OS7_DpOS8_ at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/type_traits:3840
@ 0x387381 _ZNSt3__116__thread_executeINS_10unique_ptrINS_15__thread_structENS_14default_deleteIS2_EEEEMN4kudu16MasterStressTestEFvvEJPNS6_26MasterStressTest_Test_TestEEJLm2EEEEvRNS_5tupleIJT_T0_DpT1_EEENS_15__tuple_indicesIJXspT2_EEEE at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/thread:281
@ 0x386acb std::__1::__thread_proxy<>() at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/thread:?
@ 0x2b11b0 __tsan_thread_start_func at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:959
I20260810 08:00:40.727821 15439 ts_tablet_manager.cc:1428] T 5990e7ea243843bb869de5dde3e6cd23 P 3fc5e568f3724f158b616c303efb9eae: Time spent starting tablet: real 0.021s user 0.005s sys 0.001s
I20260810 08:00:40.728691 15439 tablet_bootstrap.cc:492] T 389b30cba2b149e9ab0de8c191808db2 P 3fc5e568f3724f158b616c303efb9eae: Bootstrap starting.
I20260810 08:00:40.713241 15490 consensus_queue.cc:260] T 3593a8e38e1a44f68a54bd52b19a3c6d P 9d7219686f304f62a54acc7497cbbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:40.739079 15490 ts_tablet_manager.cc:1428] T 3593a8e38e1a44f68a54bd52b19a3c6d P 9d7219686f304f62a54acc7497cbbd07: Time spent starting tablet: real 0.051s user 0.007s sys 0.000s
I20260810 08:00:40.739964 15490 tablet_bootstrap.cc:492] T a59c73c8a95f4230946357b8aa58a1b3 P 9d7219686f304f62a54acc7497cbbd07: Bootstrap starting.
W20260810 08:00:40.744125 15234 consensus_peers.cc:487] T 8364045e3fa2441e8dc1a1d87f73361d P 3fc5e568f3724f158b616c303efb9eae -> Peer a91b27762182454baada71e91ceab224 (127.14.57.193:43601): Couldn't send request to peer a91b27762182454baada71e91ceab224. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8364045e3fa2441e8dc1a1d87f73361d. This is attempt 1: this message will repeat every 5th retry.
I20260810 08:00:40.744930 15490 tablet_bootstrap.cc:654] T a59c73c8a95f4230946357b8aa58a1b3 P 9d7219686f304f62a54acc7497cbbd07: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:40.749394 15439 tablet_bootstrap.cc:654] T 389b30cba2b149e9ab0de8c191808db2 P 3fc5e568f3724f158b616c303efb9eae: Neither blocks nor log segments found. Creating new log.
I20260810 08:00:40.754181 16127 raft_consensus.cc:686] T adada21bc1bf4f19a6347d0935bd74bd P 3fc5e568f3724f158b616c303efb9eae [term 1 LEADER]: Becoming Leader. State: Replica: 3fc5e568f3724f158b616c303efb9eae, State: Running, Role: LEADER
I20260810 08:00:40.755195 16127 consensus_queue.cc:237] T adada21bc1bf4f19a6347d0935bd74bd P 3fc5e568f3724f158b616c303efb9eae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:40.721431 16089 catalog_manager.cc:669] Loaded metadata for table default.Table_d797c9d1654e4f828ed4b8a7cce2460d [id=3b848cb6be6b40ca979e8daf9bb16591]
I20260810 08:00:40.706633 15890 rpcz_store.cc:276] 0810 08:00:36.482346 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:36.489999 (+ 7653us) service_pool.cc:227] Handling call
0810 08:00:40.705944 (+4215945us) ts_tablet_manager.cc:1145] Deleting tablet 1ce493c78a30404ca15e5b2568446bdf
0810 08:00:40.705986 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:40.706250 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4214890}
W20260810 08:00:40.769591 14681 catalog_manager.cc:4821] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet 1ce493c78a30404ca15e5b2568446bdf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ce493c78a30404ca15e5b2568446bdf already in progress: creating tablet
@ 0x7f3ef70d16db start_thread at ??:0
I20260810 08:00:40.772882 15980 raft_consensus.cc:482] T a4a78b2a960d4d78aa1f48a9c9edf3be P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:40.773355 15980 raft_consensus.cc:504] T a4a78b2a960d4d78aa1f48a9c9edf3be P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:40.775429 15980 leader_election.cc:288] T a4a78b2a960d4d78aa1f48a9c9edf3be P 9d7219686f304f62a54acc7497cbbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91b27762182454baada71e91ceab224 (127.14.57.193:43601), 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729)
I20260810 08:00:40.777307 15980 raft_consensus.cc:482] T b18980bffe6045b5b69bcecb47b4c711 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 08:00:40.777766 15980 raft_consensus.cc:504] T b18980bffe6045b5b69bcecb47b4c711 P 9d7219686f304f62a54acc7497cbbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:40.780438 16089 catalog_manager.cc:669] Loaded metadata for table default.Table_7d8f3d372c5c447681edeea50da2a8d8 [id=4333f028b97f494cac962fe1607cb67d]
@ 0x7f3ef247c61f clone at ??:0
I20260810 08:00:40.782009 16089 catalog_manager.cc:669] Loaded metadata for table default.Table_64e1c230e9624fc9ba6c328346150ffe [id=46e6c2115d8c4b5e9907ed76b273638a]
I20260810 08:00:40.830821 16089 catalog_manager.cc:669] Loaded metadata for table default.Table_187379f079bf4283824c6b02b86271ba [id=4abdfd680cf74e3791c0389daf6d4e43]
I20260810 08:00:40.832480 16089 catalog_manager.cc:669] Loaded metadata for table default.Table_a89c79d2a99b415cbac968e3e8a9d53d [id=4c70468b9e8649849cd0e79d59be6bb9]
I20260810 08:00:40.786854 14988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b18980bffe6045b5b69bcecb47b4c711" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:40.836397 16089 catalog_manager.cc:669] Loaded metadata for table default.Table_1782c242ad164aa69cb679bbb072934e [id=5efc99cbf96549d681a989f1886b65f2]
I20260810 08:00:40.841120 16089 catalog_manager.cc:669] Loaded metadata for table default.Table_c4fc713ca0b54050b0475b8b746658f7 [id=69d89f0f5a4e4e8a8814ea01bca243ce]
I20260810 08:00:40.786868 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4a78b2a960d4d78aa1f48a9c9edf3be" candidate_uuid: "9d7219686f304f62a54acc7497cbbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91b27762182454baada71e91ceab224" is_pre_election: true
I20260810 08:00:40.811553 15890 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34630 (request call id 187) took 3720 ms. Trace:
I20260810 08:00:40.841711 15890 rpcz_store.cc:276] 0810 08:00:37.070704 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:37.071092 (+ 388us) service_pool.cc:227] Handling call
0810 08:00:40.770725 (+3699633us) ts_tablet_manager.cc:1145] Deleting tablet 8b5a629a7f8041179a006fa412fb6b48
0810 08:00:40.786461 (+ 15736us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:40.790632 (+ 4171us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3698528}
W20260810 08:00:40.843501 14681 catalog_manager.cc:4821] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet 8b5a629a7f8041179a006fa412fb6b48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8b5a629a7f8041179a006fa412fb6b48 already in progress: creating tablet
I20260810 08:00:40.851447 15890 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34630 (request call id 188) took 3564 ms. Trace:
I20260810 08:00:40.852016 15890 rpcz_store.cc:276] 0810 08:00:37.286589 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 08:00:37.288296 (+ 1707us) service_pool.cc:227] Handling call
0810 08:00:40.844442 (+3556146us) ts_tablet_manager.cc:1145] Deleting tablet 33cacf1276d44b24abe6aff12ea8ea51
0810 08:00:40.851068 (+ 6626us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 08:00:40.851377 (+ 309us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3497698}
W20260810 08:00:40.853330 14681 catalog_manager.cc:4821] TS a91b27762182454baada71e91ceab224 (127.14.57.193:43601): delete failed for tablet 33cacf1276d44b24abe6aff12ea8ea51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33cacf1276d44b24abe6aff12ea8ea51 already in progress: creating tablet
I20260810 08:00:40.856491 15437 tablet_bootstrap.cc:492] T 8a8454f693494ab3b82f5b85c2d62bb2 P a91b27762182454baada71e91ceab224: No bootstrap required, opened a new log
I20260810 08:00:40.856940 15437 ts_tablet_manager.cc:1397] T 8a8454f693494ab3b82f5b85c2d62bb2 P a91b27762182454baada71e91ceab224: Time spent bootstrapping tablet: real 0.307s user 0.040s sys 0.050s
I20260810 08:00:40.849512 16089 catalog_manager.cc:669] Loaded metadata for table default.Table_1dcf09dc44214dae8e1ed6319bdc825f [id=6ab4c475469c438888e3efe0f7a59b3f]
I20260810 08:00:40.863154 16089 catalog_manager.cc:669] Loaded metadata for table default.Table_4a73c802a93940dea0793cc5e71e1f6f [id=6b0af0270f5d40e1a37b309b130be42a]
I20260810 08:00:40.864750 16089 catalog_manager.cc:669] Loaded metadata for table default.Table_19d382281f324b2e82afd3247cebd644 [id=6bc3360c711f4209975c7f731892dfbb]
I20260810 08:00:40.861908 15437 raft_consensus.cc:348] T 8a8454f693494ab3b82f5b85c2d62bb2 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:40.864590 15890 tablet_replica.cc:330] T 1f427f4093e5417a9ca79b1805b0675e P a91b27762182454baada71e91ceab224: stopping tablet replica
I20260810 08:00:41.187062 16070 raft_consensus.cc:482] T 1f427f4093e5417a9ca79b1805b0675e P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fc5e568f3724f158b616c303efb9eae)
I20260810 08:00:41.210196 15437 raft_consensus.cc:374] T 8a8454f693494ab3b82f5b85c2d62bb2 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 08:00:41.210510 15437 raft_consensus.cc:729] T 8a8454f693494ab3b82f5b85c2d62bb2 P a91b27762182454baada71e91ceab224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91b27762182454baada71e91ceab224, State: Initialized, Role: FOLLOWER
I20260810 08:00:41.120095 16114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38a7164890624423a7427871aa2b10e0. Latest consensus state: current_term: 4 leader_uuid: "38a7164890624423a7427871aa2b10e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } } }
I20260810 08:00:41.211242 16114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [sys.catalog]: This master's current role is: FOLLOWER
I20260810 08:00:41.211194 15437 consensus_queue.cc:260] T 8a8454f693494ab3b82f5b85c2d62bb2 P a91b27762182454baada71e91ceab224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } } peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } }
I20260810 08:00:41.212924 15437 ts_tablet_manager.cc:1428] T 8a8454f693494ab3b82f5b85c2d62bb2 P a91b27762182454baada71e91ceab224: Time spent starting tablet: real 0.356s user 0.005s sys 0.001s
I20260810 08:00:41.202586 14963 tablet_service.cc:1514] Processing DeleteTablet for tablet 5990e7ea243843bb869de5dde3e6cd23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 08:00:32 UTC) from {username='slave'} at 127.0.0.1:34630
I20260810 08:00:41.210510 16070 raft_consensus.cc:504] T 1f427f4093e5417a9ca79b1805b0675e P a91b27762182454baada71e91ceab224 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc5e568f3724f158b616c303efb9eae" member_type: VOTER last_known_addr { host: "127.14.57.195" port: 37729 } } peers { permanent_uuid: "a91b27762182454baada71e91ceab224" member_type: VOTER last_known_addr { host: "127.14.57.193" port: 43601 } } peers { permanent_uuid: "9d7219686f304f62a54acc7497cbbd07" member_type: VOTER last_known_addr { host: "127.14.57.194" port: 36265 } }
I20260810 08:00:41.125234 16140 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 38a7164890624423a7427871aa2b10e0)
I20260810 08:00:41.215286 16140 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e8cadb26fe0428a964f66ef8172f3be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f2d868d76c440f8443b33e1cd19f58" member_type: VOTER last_known_addr { host: "127.14.57.254" port: 43757 } } peers { permanent_uuid: "38a7164890624423a7427871aa2b10e0" member_type: VOTER last_known_addr { host: "127.14.57.253" port: 40439 } } peers { permanent_uuid: "1e8cadb26fe0428a964f66ef8172f3be" member_type: VOTER last_known_addr { host: "127.14.57.252" port: 33819 } }
W20260810 08:00:41.219830 14918 leader_election.cc:341] T 1f427f4093e5417a9ca79b1805b0675e P a91b27762182454baada71e91ceab224 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3fc5e568f3724f158b616c303efb9eae (127.14.57.195:37729): Not found: Tablet not found: 1f427f4093e5417a9ca79b1805b0675e